When AI Meets Network Observability with Avi Freedman, CEO at Kentik

(Intro Narrator at 00:00:00) Today, we're talking to Avi Freedman, CEO at Kentik, about the intersection of network observability and AI, how they're tracking cyber attacks in Ukraine, and Avi's time in Japan. You're listening to Joel Beasley, Modern CTO.

(Joel Beasley at 00:00:20) I was curious for you to continue and tell me what are you doing in Asia?

(Avi Freedman at 00:00:26) So we have probably a couple dozen customers in Asia. There is a conference, JANOG, the Japanese Network Operator Group Conference, that is the week before the Pacific Telecom Council conference that happens in Hawaii. And I decided to go to both conferences and wound up getting to neither, but it was an excuse to start scheduling an Asia trip. We had too many customer meetings, so we did a couple days in Seoul beginning of last week, where I learned that I was mispronouncing Taekwondo, which I practice. I sort of suck at it, but I've been doing it. And a little over a week in Japan, and we have a bunch of customers there as well. And my wife and I like going to Japan, and then Hong Kong, where my wife has not been back to and decided to do a few days while we were here. I just decided we're getting older, and once I adjust to the time zone, might as well stay over here for a little bit longer.

(Joel Beasley at 00:01:30) And where's your permanent residence?

(Avi Freedman at 00:01:33) In Seattle. I'm from Philadelphia.

(Joel Beasley at 00:01:38) Very cool. And so what's your favorite thing to do over in Japan?

(Avi Freedman at 00:01:44) Well, that would have to be eating. And most of the kinds of Japanese cuisine, I love. And that's probably 80% of my Japanese is food. 15% is Shotokan karate words, and 5% is don't touch my mustache and all the 20 words that you learn to grease social lubrication. Second might be sumo, although we didn't go to sumo even though it is on. I just didn't have time this trip. But they actually have bleacher seats, which was exciting, because in July, we were there, and we went to sumo in Nagoya, and we had to sit tatami. We didn't have to sit seiza where you sit on your knees, but we had to sit tatami. And my American body is not designed really to do that. So we sort of cheated and got these little footstools at the Daiso and sort of sat on the footstools. The two inches of elevation helped a lot. But so it's fun to see the sumo. So much more intimate stadiums than we have in the US.

(Joel Beasley at 00:03:01) I heard from a friend that the technology there is crazy when you're walking around the streets and the vendors and all of that. That is just incredibly advanced technology. Is that true?

(Avi Freedman at 00:03:11) So I think that used to be even more true. I'm not sure that's as much true, but Libby, who runs People for Kentik, lived in Japan for a number of years and taught. And I said to her this morning, do you notice how quiet it is? In the United States, sometimes when you go into a food hall or even when you go into the convenience store, Family Mart, 7-Eleven, Lawson's, everything is talking to you and beeping. Every time the door is open and everyone's, or just as you're walking by all the stands and the food halls, it's all very sonorous in its own way, but it's definitely a different experience. But now that we have Google Maps and electronic payment and these things that used to, in Korea and Japan, there used to be cell phones for that. I used to go to Akihabara Electric Town and get Sony VAIO laptops that Dynamism used to import for $5,000, and it was cheaper to go to Japan and get them. But Apple makes some really good laptops with high resolution, good battery, and they run Unix. So what more do you want?

(Joel Beasley at 00:04:22) I want the Vision Pro. That's what I want.

(Avi Freedman at 00:04:25) Okay. Well, you can have that. I guess that'll be an interesting one. I'm sort of following from a distance smart glasses and VR and the intersection there. And hopefully, no one will be beat up for being pretty creepy with those. So, we'll see.

(Joel Beasley at 00:04:44) So how do I pronounce the name? Is it Kintik? Is that how I say your company name?

(Avi Freedman at 00:04:47) Kentik, K-E-N-T-I-K.

(Joel Beasley at 00:04:49) Okay. And what is it exactly that you guys are doing?

(Avi Freedman at 00:04:53) So network observability. We take telemetry from the infrastructure layer that's in and around the network, and we make it usable for people. So when an Uber driver asks me what does my company do, I say we make the internet go. Or next version longer, we make the internet, we make networks fast and secure. And so our customers are IT service providers, web companies, people that deliver their packets as their revenue, people that the network doesn't go, the company doesn't go. Those are the kind of folks that are direct buyers, and then we have other users in the company and security and other operations and work with other observability players, but that's primarily what we do.

(Joel Beasley at 00:05:37) All right. So are you the founder of the company?

(Avi Freedman at 00:05:39) I am a founder, and I run the company. Yes.

(Joel Beasley at 00:05:42) How did that come to be? How did the company start?

(Avi Freedman at 00:05:46) Well, I was at a company called Akamai, large content distribution network for about 10 years, and did a bunch of federal work. And after I left Akamai, some people asked me to build, you ever see Silicon Valley, the show?

(Joel Beasley at 00:06:03) Oh, yeah.

(Avi Freedman at 00:06:05) They wanted me to build the box. So I built the box. Not the same logo as Silicon Valley, but I built the box, which could see packets and run a bunch of applications. And everyone that bought the box was like, what do we do with all this data? I was like, well, that's a much more interesting problem because I started an ISP in 1992, and I was at a big backbone after that, and I was at Akamai, which is recurring revenue, and I've always been into recurring revenue. And so decided there's some hard problems there and talked to a bunch of people, and they said, yeah, we need help running our infrastructure. We don't really have good dashboards and driving a car, sort of with some speed indicators and stuff like that and not really understanding what's going on for their infrastructure. And then when I said, would you be willing to send me data so I could mock this up on my Usenet infrastructure, they're like, oh, we'd rather have it as a service. We know it's a ton of data. We wouldn't want to actually run this. So it's like, okay, that seems aligned. So we started the company 10 years ago, 2014, mocked it up, and got a bunch of people using it and raised venture capital later 2014 and then started going.

(Joel Beasley at 00:07:15) Share with me more if you can. I'm just interested in the partnership aspect of you founding the company with other people. How did that go about?

(Avi Freedman at 00:07:25) So in hindsight, whether it was in partnership or not, another interesting thing we can talk about. I wish I had bootstrapped a little longer, because what is now called product-led growth is sort of harder to get religion about once you actually start growing and have massive customers that you can use people to leverage onboarding complex infrastructure, things like that. Whereas if you decide to bootstrap, not just from an economic perspective, but from an onboarding perspective, you have to seek that elegance that the lazy, the outstanding architects and developers are lazy. They seek the elegance because they don't want to do things over and over. And I wish I had done that and it might've been with the same partners, but I had a friend who I actually introduced to run operations at Cloudflare. I had known him for some time. I had worked with him on some projects before. And he was like, oh, you should move out to Silicon Valley and whatever this idea that you want to do, you should start it out there. He introduced me to, they're both named Ian. Ian Pye, who was the first employee at Cloudflare who I'd waved hello to because long time ago, I gave him their first servers. And then, again, my friend, Ian Applegate, referred there to run operations. And we were talking and decided to start the company. And then before we raised venture capital, a friend of mine introduced me to someone that he thought I would like, who thought I was still in the internet infrastructure industry, and I thought he was in the internet infrastructure industry. But he joined to lead the sales effort at Kentik, which we weren't necessarily looking for at the time, but in hindsight has been really great. So not quite Silicon Valley, the show, although it was contemporaneous with us. It was sort of amusing in a few different ways.

(Joel Beasley at 00:09:31) So you said you know the one of the first employees at Cloudflare?

(Avi Freedman at 00:09:37) Yes. I guess it can be ambiguous to say partner nowadays, but I'm old, so business partner. You use the word partner, business partner. Yeah. Ian Pye, he was the first employee. He was Lee's office mate, I believe, at Santa Cruz. And Ian took my, well, I sometimes say beautiful C code, he would probably argue that, but he took my C code and rewrote it into Go. And he promised me that Go was okay, because I said, oh, can you do static memory allocation if you want to? Can you have to have a garbage collector? Oh, yeah, you could absolutely do it, which we did later on by writing a slab allocator. So you could, but it wasn't actually easy. I said, can you make static binary? And it took us four years to figure out how to make static binaries for our agents with Go. But we figured it out. So people like Go. I'm not one of those people. I don't think that uppercase should be scope. But I don't like Python, and that's the network language because I don't think that white space should be syntax. So, I'm an old curmudgeon, and it doesn't matter because I don't write the code for the company.

(Joel Beasley at 00:10:50) Do you have any current favorite authors that you learned a lot from when you were writing code?

(Avi Freedman at 00:10:56) Who wrote C Traps and Pitfalls? That was a good book. I just saw it on my bookshelf. When I'm at home, I have a bookshelf of my '90s favorites. The Lions book?

(Intro Narrator at 00:11:07) That was Andrew Koenig.

(Avi Freedman at 00:11:10) That was a really good book. Andrew Koenig, C Traps and Pitfalls.

(Joel Beasley at 00:11:17) You like Martin Fowler at all?

(Avi Freedman at 00:11:19) I've met Martin Fowler. I do like him. I'm a little bit less, some of the broad pattern thinking is just not, it's not my jam. I'm more into low level distributed system type stuff, and a lot of the stuff he writes about is a little bit higher level than that. My favorite essay from Mythical Man-Month is not the how many, adding birthing people doesn't make a baby faster. It's actually there's probably a reason why the cruft is there. I forget what the title of that essay was, but it basically says, unless you think, when you say, oh my God, this code is so horrible, we must immediately rewrite it, not even looking at the code and understanding why it's horrible. I feel like that with the internet too. It's like, oh, let's replace all the routing protocols. Do you have any idea why they're working and not working and why? Again, I feel like, get off my lawn, Statler and Waldorf up in the, we tried that in the 1300s, and the internet didn't work. We crashed the internet. But I think there's some truth to that also. You gotta be careful, because there's probably some hack you did for a customer because of their bug, but you're gonna break a $3 million a year customer if you don't incorporate it somehow.

(Joel Beasley at 00:12:49) Yes. I'm getting the, I've got early onset curmudgeon. I'm 35. I'm about to be 36, and I've got three kids. And I just started to get the inklings of the, hey. But it's partly maturity. It's when you're younger, you're like, let's, we just gotta do it the best way possible. It's like, rip it down and just rebuild it the best. And then you realize that, oh, there's a lot of reasons why these institutions, why these systems are in place. And we can't just rip everything out. And then that breeds the maturity of value and delivery of value. Right? Because then it's like, okay. Well, if we're just gonna be, if we know we need to be useful to earn. And so how can we be useful? And then how can we achieve that without breaking other things and be very direct with it? And so that's where I'm at currently.

(Avi Freedman at 00:13:34) Innovation is awesome. But I also think that it's important to understand things that were tried and why they worked or didn't work. And you could disagree with whether they worked or not and why they worked or didn't work, but it's important to try to understand. I saw this when we were at Akamai and we tried to get into ad tech. And Akamai is full of super smart people, super, super smart people. But when you go into an industry and you're super smart, but you have no idea why anything is the way it is, people know. And it's actually important. So if you're building a database, it's important to know what to throw out. That's important to understand that some of the stuff is important for certain reasons. If you want to deselect it. And we did when we built Kentik. We're like, or it was called Cloud Helix and Hydra SQL. And actually, I forget what I called it before Ian named it Hydra SQL. But it was like, we're not gonna do that because we're gonna be append-only. But it was like, you try to figure out these constraints and do wishful architecture that allows you to get started and say, okay. Well, but we've architected it so we can go do that. It's like one of my favorite content marketing sites is enterpriseready.io, I think, from Replicated. And it's like, oh, you little SaaSlets. You wanna start a SaaS company. Here's the shit you need to think about. You don't need to build it. You just need to realize at some point, you're gonna need to do SSO. You're gonna need to do role-based access control, RBAC. You're gonna need to do some sort of logging in and access to that for the enterprise. There's a whole bunch of stuff that you're gonna need to do, and it turns out it's a lot easier even if you're not gonna build it if you don't build inconsistent with things you know you're gonna need to build. So it can be helpful.

(Joel Beasley at 00:15:29) So I was reading an article about Kentik, and they talked about how you were observing a cyber attack in Ukraine. Can you tell me about that?

(Avi Freedman at 00:15:40) Well...

(Joel Beasley at 00:15:41) We're allowed to talk about that?

(Avi Freedman at 00:15:43) Yes. We're allowed to talk about that. It turns out that a lot of those cyber attacks are really more like kidnappings almost. It's like, DDoS is something, DDoS detection is something that more than half of our customers use Kentik for.

(Avi Freedman at 00:16:02) So real-time distributed, seeing all the telemetry in real time. Where are these attacks? Are they misconfigs? Is it the application? Is someone pounding your API, or is it someone from the outside just blasting you with traffic? What should I do about it? That's a big use case for us for folks that are Internet-focused.

(Avi Freedman at 00:16:20) And a lot of these cyber attacks are just volumetric. So in some sense, DDoS is like, you know, I shove your head in a toilet, and that makes it easier for me to pick your pocket. But is it really security? Like, I get into your databases and I hack all your stuff?

(Avi Freedman at 00:16:38) You know, I believe that incident was an interesting kind of attack coming in against both infrastructure and, you know, effectively infrastructure sites and users in Ukraine. And unfortunately that stuff happens an awful lot. Not only that, but another thing that we track that Doug Madory tracks at Kentik is just suppression of the Internet in certain regions when politically, local, I guess you can say regimes, governments, don't want people to be communicating at that time. And that, you know, it's still happening all over the world in various ways.

(Avi Freedman at 00:17:25) I'm sorry that my—

(Joel Beasley at 00:17:26) It's okay.

(Avi Freedman at 00:17:28) Master lights just went off.

(Joel Beasley at 00:17:29) It's alright. Your wife needs to go to bed. Yeah. So tell me, like—so you can abstract it to some made up government name, but how does the government actually suppress parts of the Internet specifically?

(Avi Freedman at 00:17:47) Well, in a lot of countries—I mean, you know, there's a lot of countries in the world—and in an awful lot of them, it turns out that the ISPs have to do whatever the government says. So if they say, you know, in some countries there's filters in place for layer four through layer seven content. In some places, there's infrastructure to push layer three like black holes and things that the government tells you to do. And in some cases, it's find the on/off switch and just go, okay.

(Avi Freedman at 00:18:20) And then all of a sudden the cell towers go off or the data to the cell towers go off, or the default route, if you know what that is, goes away. And you can only get internal to certain networks or applications. Sometimes it's DNS filtering. Sometimes it's, you know, different layers. And in an awful lot of countries—I mean, dozens and dozens and dozens, probably more than half the countries in the world—it's, you know, it may be deregulated, but it's tightly controlled.

(Avi Freedman at 00:18:50) You know, some of the ministry of truth, communications, packets, bits, whatever, says make it stop, then they make it stop to whatever their technical capability. And that could be regionally, that could be at a layer, that could be blocking IP addresses or DNS. You know, it's just different per different countries.

(Joel Beasley at 00:19:09) That's crazy. Do we do that here in America?

(Avi Freedman at 00:19:14) Not like that. No. Okay. No, we don't.

(Avi Freedman at 00:19:19) I'm sitting actually in China right now. Well, in Hong Kong. Yeah. So that's different. But no.

(Avi Freedman at 00:19:27) In America, the government does have certain abilities to subpoena information, you know, and get court orders. And that's been going on for a very long time. And, you know, it's a regulatory framework. But no, they can't say make this stop and then it will just stop. Again, they might be able to get a subpoena to instruct them to turn off a user, but not a city.

(Avi Freedman at 00:19:51) Now I don't want to get too much into politics.

(Joel Beasley at 00:19:56) Well, Twitter Files is an event that happened.

(Avi Freedman at 00:20:01) Yeah.

(Joel Beasley at 00:20:01) It was covered by both—but the data was given access to both a variety of political people or writers that had a variety of political events. So it was covered in a pretty good scale. But it was definitely both sides.

(Joel Beasley at 00:20:17) Both sides of the government in the United States, both parties were doing things to prohibit content from being—to censor content.

(Avi Freedman at 00:20:26) So now there's multiple layers. Right? You're familiar with the network stack, you know, and transport. And so people argue, is politics layer zero or layer eight, right, or layer nine, whatever?

(Avi Freedman at 00:20:41) I'm like, where is the politics in that? But I would say it's up the stack. Right? Because it's the content. Right?

(Avi Freedman at 00:20:48) So I think there's infrastructure. And I would say even up through layer seven, like infrastructure, that's proxies, that's doing things and down is one thing. Putting financial like boycotts or legal or regulatory pressure or political pressure or marketing pressure to show or not show something—that's like above my pay grade. It's above the Internet.

(Joel Beasley at 00:21:18) Yeah.

(Avi Freedman at 00:21:18) That's above the infrastructure. So but if you just look at the infrastructure side of it, including proxying it all the way down, no. That's not something that we've done. But there's definitely people, some of my libertarian friends are concerned that some of the rights that the government asserted, or it was talking about asserting, you know, as we were figuring out the COVID response, were awfully close to, you know, hey, maybe we should. And to your point, if you go way above the infrastructure, what is harmful speech? And again, that's a whole separate thing.

(Joel Beasley at 00:21:55) Yeah.

(Avi Freedman at 00:21:56) And I sort of view it as above the infrastructure. So—

(Joel Beasley at 00:22:00) Okay. So in other countries, I guess to circle back to the beginning of this, in other countries they're messing with the lower—all the layers. They'll just shut pipes off. Yeah. Turn entire things off. Yeah.

(Joel Beasley at 00:22:11) We're not doing that here. Okay.

(Avi Freedman at 00:22:13) No. No. No. And we don't have, you know, sort of like the great firewalls which some countries have. You know, I've had people tell me, oh, this network copies all their packets.

(Avi Freedman at 00:22:24) I'm like, I know they don't because I know before, you know, they were a customer, they had basically no visibility. Like, I know what their—I know what their infrastructure is for seeing what their network is, and I know there's no great packet taps running on, you know, American infrastructure.

(Joel Beasley at 00:22:42) Do you think that Starlink is going to pose a problem to these countries' ability to impose these restrictions?

(Avi Freedman at 00:22:50) You know, I think it's an interesting question. Starlink is much closer to infrastructure than VPNs, but VPNs have, you know, been an area of concern for these countries. In fact, India has basically said, like, you cannot run VPNs in our country unless, you know, unless you give us the keys. And so most VPN providers have pulled out of there. You know, Starlink so far, I think there's been some political, you know, are they serving certain areas or not?

(Avi Freedman at 00:23:20) But I think so far, that's been at the level of sort of marketing and pressure and not regulatory, but I could be wrong. I'm not an expert on that.

(Joel Beasley at 00:23:29) That's fascinating to me.

(Avi Freedman at 00:23:30) Yeah. I mean, it's definitely an important part of the world. And, you know, a lot of people, like, I flew here on a flight, didn't have Wi-Fi, and I was like, well, you know, that's, okay. You know, I'll read a book. I've got Kindle.

(Avi Freedman at 00:23:45) But I've talked to people in the last few months who are like, it's horrible. It's a crime. I don't have Wi-Fi. I'm flying 30,000 feet at, you know, 600 miles an hour and I don't have Wi-Fi. Like, how could this happen?

(Avi Freedman at 00:23:59) But, you know, I've had enough time remembering when that wasn't the case that I guess it seems more amazing to me that we have it. So, but, yes, it's definitely scary. There's countries—you may be following where the infrastructure is definitely being put into place that you might not be able to travel, you might not be able to bank if you have unpopular opinions. So, you know, it's scary what technology can do, and it's great what technology can do.

(Joel Beasley at 00:24:35) Yeah. Drive people towards systems that will allow them to, and then those systems will—

(Avi Freedman at 00:24:42) I'm a big fan of decentralized and—Okay. You know, on the other hand, you actually have to be, for example, part of banking, and you have to be part of some of these systems. So, but how can you do that as much decentralized as possible is, you know, it's good to have options at least.

(Joel Beasley at 00:25:00) Yeah. Yeah. Yeah. We're having, and Josh, correct me if I'm wrong. Is it David Johnson?

(Joel Beasley at 00:25:06) He coined the phrase DApps. Or he's working on a new project. I found him through a friend of a friend, and it was called Morpheus, mor.org. And it's essentially decentralized AI, so decentralized GPT that can make it—it just makes it more accessible, so I can actually download it. It can run locally. It could be decentralized, and I can have it do things like I could say, you know, hey, go buy this or make this type of transaction, and it helps with the making those types of transactions more accessible to people that don't necessarily know how to do it in a technical way.

(Joel Beasley at 00:25:42) Because right now, most people that don't know how to do it in a technical way, they're buying through Coinbase or whatever exchange, and this is giving that power to them to have an agent running on their computer that can do this stuff for them. And so I thought that was a pretty interesting project. So we're doing a special episode on that in like the next two weeks.

(Avi Freedman at 00:26:01) You know, this is a little bit separate, mostly separate from, you know, sort of what I'm doing, you know, my day job at Kentik, but it's really interesting. We talked about software engineering and Mythical Man-Month and trade-offs. Right? What people say versus what people do. When there's a conflict between convenience and privacy, what percentage of people will actually go for, you know, the privacy?

(Avi Freedman at 00:26:36) And so, you know, how do you make it maximally convenient? But then again, you know, I think more about DApps and like communication and how do you do that with Reddit and Twitter, you know, all those things. But, and, you know, I'm a Usenet fan from way back. You know, and obviously, we figured out how to do chat and these other things. But, you know, there's advantages to centralized.

(Avi Freedman at 00:27:01) Right? I mean, I use Twitter a lot still to consume information. I try using Usenet, but there's too much garbage there, so I don't do that. And I just don't have enough time, generally. So I spend most of my time on RSS feeds catching up 30 days behind before Feedly marks all of them as read.

(Avi Freedman at 00:27:17) So—

(Joel Beasley at 00:27:19) I like that, convenience versus privacy, because the more convenient it is, the harder it is to get the privacy. Typically, privacy is something you have to go seek out.

(Avi Freedman at 00:27:30) Yes. And it's unfortunate, but there's a whole bunch of things that we need to sort of train people to do. Right? Even before AI, you know, just with Wikipedia and everything else, it's like, amazing what we can get on the Internet, but you need to have a good bullshit filter. Right?

(Avi Freedman at 00:27:47) You need to be able to sense emotion words and agendas and all those things. And so I think that should be a key, like, you know, bullshit filter 101, 102, 201, 202. Like, it should be like a high school track. You know, taking control of your own privacy to the extent you want to. You know, it's unfortunate, but if you're not into technology or, you know, and if to participate in the kind of things that everyone else does, you know, it's really hard to do that.

(Avi Freedman at 00:28:20) You know, it could be tough. Whether that's communication or, you know, commerce or whatever. So the good news is there's a lot of people passionate about it and innovating, and we're just at the beginning. So—

(Joel Beasley at 00:28:32) Have you looked at the Unstoppable Domains project?

(Avi Freedman at 00:28:37) I have probably—the Web3 domains? Yes. I mean, I have some DNS domains, but that's not the—that's not the Ethereum Name Service. Unstoppable was something else. Right?

(Avi Freedman at 00:28:53) I'm fuzzy on it. It's been a few years.

(Joel Beasley at 00:28:55) Yeah. I think they used it. I haven't—I'm not an expert in this area either. My—it was a weekend rabbit hole is what I call it.

(Avi Freedman at 00:29:02) Oh, okay.

(Joel Beasley at 00:29:02) I went on a weekend rabbit hole of Unstoppable Domains, Web3 domains, and basically how they're—the way that they're working in the sense that you can't go get a court order subpoena and take it to GoDaddy or ICANN or whatever it is and just stop the domain if you don't like what's happening on the domain.

(Avi Freedman at 00:29:20) Mm-hmm.

(Joel Beasley at 00:29:21) Right? And so that single utility was interesting to me. And so I started saying, like, oh, well, if I want to have a domain like this, how is it going to, you know, work? And that's kind of where I got a little fuzzy is if they own the lower layers, can they not just stop you from accessing those domain extensions. Right?

(Avi Freedman at 00:29:45) Well, yeah. I don't want to—I mean, we can go down the crypto rabbit hole as much as you'd like. And I'm a little fuzzy on some of it because I haven't looked at the domain stuff, which is embarrassing because I am actually semi-official—and the Minister of Bits for Sealand, which is a principality in the North Sea. And in partnership with them, we did HavenCo, which is putting servers out in the North Sea over 20 years ago. And it turns out that should have been decentralized, but the tech wasn't there for it at the time.

(Avi Freedman at 00:30:23) But they sell—turns out non-fungible entities like territory and titles and things like that. So we've looked a lot at crypto and things like that. And, you know, you can hold in your head two truths. One of them is an awful lot of blockchain stuff could be a MySQL or SQLite database, you know, with an API and a couple of Amazon regions.

(Avi Freedman at 00:30:54) Because as you said, like, yeah, it's theoretically the blockchain, but if you control the interfaces to it, you could actually censor it as much as you want. Right? On the other hand, there's some stuff that, you know, is actually pretty—could be pretty practical if it were just faster, like Filecoin, IPFS-type stuff. And there's other stuff that, you know, maybe, as much as I like the idea, and this is something we debated for HavenCo for Sealand rather of, you know, if they ever did citizenship, you know, how much of it would be tied to economics and how much of it would be tied to unique personhood, scanning your eyeball.

(Avi Freedman at 00:31:33) And the way they implement it, I'm not sure that's the—

(Joel Beasley at 00:31:33) I'm not sure that's the right way to do it either. So—

(Joel Beasley at 00:31:36) Yeah. The InterPlanetary File System, that thing to me is so—Filecoin and storage are like 80% of my crypto holdings because—

(Avi Freedman at 00:31:44) And Sia and MaidSafe—and do you have any Chia, which is—no. Chia makes me angry. Bram's a brilliant guy. But filling up hard disks with nothing and not storing anything on them? Come on.

(Avi Freedman at 00:31:54) That's not—I mean, it's greener than proof of work, but not that much. But, yeah. I think the problem, the challenge is, all of them are really good. I would just use Usenet. I mean, Usenet's a third of a petabyte a day, and it's fast.

(Avi Freedman at 00:32:10) And it's—I mean, it's less decentralized than when there were 100 people doing full feeds because it's 30 gigabits a second, but there's a lot of people doing it. You can post and get it. I know there's been work. I know Cloudflare and other people have built IPFS gateways, and I think the innovation is great. I'm a little frustrated they took $500 million and, I don't know, what, five years ago.

(Avi Freedman at 00:32:31)
I expect some more to come out of it. But I think the challenge—and I think storage has made some progress towards this—is you need fast and reliable. Like, it's okay to have it be like a mesh and sort of permanent and available, but you need it to be fast and reliable or it's this privacy and security versus convenience. Look up Tahoe-LAFS. So actually, pre-Kentik, and this is sort of a goof aside project, Zooko of Zcash—I mean, I think it's still maintained, but they were doing an erasure-coded decentralized cryptographically everything file system. And we tried to get people to use it. And sort of by nature of—are you familiar with erasure coding?

(Joel Beasley at 00:33:04)
Nope.

(Avi Freedman at 00:33:04)
Okay. It's like super RAID. So think about super RAID where instead of all the disks being right next to each other, one way you could deploy it is so that you're running like RAID 6 on eight disks. So you got 10 disks, but you put five of them in each of two different places. So to get any object back, you need to get some chunks from two places. So someone stole—like, the government walked in and took all your stuff. They got nothing. Now you extrapolate that sort of the same underlying math of encoding, and you have five locations and eight disks is sufficient to pull everything back, but you only have six disks at each location. Sounds really good.

But in practice, if you deploy it like that, you'll have a buttload of latency to get anything. And so you get into these problems that some of those systems have, and I think Storj has actually done a lot of work to say, maybe it's good to actually use these technologies, but we should actually make a market at it and have some locations where it's just like S3. Like, we incent people to run it, but they're running it in data centers and it's fast enough and stuff like that. And it's interesting because you see—if you didn't follow Chia, it was really fascinating because I think it was 2020. Basically, they went from zero to 30 exabytes of people crawling around Unix surplus and buying fuse and sticking motherboards. And it was like 30 years ago at computer shows when we used to put computers together. And running NASes basically, 36 just in their living room for the promise of crypto. And then people would fill up data centers with them. I have friends that still run dozens of petabytes of Chia because they'll do something else with the hardware. Actually, they're using it for Freax. So they'll figure out they'll just do storage with them.

I just would like to see Chia be an awesome blockchain that supports, you know, ethereum-like coding on top that actually just signs objects that you store so that you're filling up these hard disks with signatures and objects, not just signatures, not just random garbage that's signed. I think that'd be pretty cool because it turns out that a lot of the Chia stuff they've incented through—I mean, people interested in hype, whatever—to be in really well-connected data centers. So you could do some really cool storage stuff. I think the challenge is if you look at Wasabi, which is a commercial S3 competitor, you look at Backblaze, which is amazing company. Again, second favorite content marketing, all their hard disk, hard drive reliability reports, now SSD. I think Backblaze has, through content marketing, pricing, nerd friendliness, a lot of different ways, their angle. I think Wasabi was going at it from bandwidth advantageousness, and Storj platform is doing it. I think it's hard to compete with S3. So you gotta think pretty carefully. Why am I doing this? What am I doing? But at the same time, if you're going to do it and it's not convenient for people to use, then you have to be a little bit of a nutcase, which I am, to use some of that technology. So I think that's something that the decentralized storage people have to crack.

I apologize. I geek about network and also storage and also systems and clustering and stuff like that. So back when I became a hobbyist, there was no sysadmin, network, programmer. It was just nerd. So we've sort of specialized.

(Joel Beasley at 00:37:19)
Yeah. We specialized into different hyper specialties.

(Avi Freedman at 00:37:19)
Yeah.

(Joel Beasley at 00:37:27)
My dad was an engineer. He learned in the Air Force, and he came out and then did odd jobs and would take me around with him to the different buildings and the projects that he had. And so I got exposure to a large variety of different types of technology from hotel systems getting their first electronic locks to the tax company getting their first computers to radio companies doing all sorts of interesting things with radio towers. And so, yeah. And back then, I was just the—I got the privilege of being born into that family. And we weren't wealthy or anything, but just because I got access to that technology, and I got to see it. And so at school, I was just the nerd at school. Like, I was the compute—and there was just nerd. It was like you could do websites, network, video games included. Like, it was all things nerd. And so, yeah.

(Avi Freedman at 00:38:22)
Yeah. It wasn't video games for me, but I was definitely a nerd. But again, contemporaneously in 1978 when I got into computers when I was eight, very fortunate family stuff. My uncle, who was of the Elmer generation of the HAM, he gave me a book on BASIC. And there weren't a lot of computers around that ran BASIC at the time, but my father happened to have some because he was a doctor, both doctors. He was a doctor doing medical research, and he happened to have a wizard who was a Unix bigot back then who got me into that. So it was just very fortunate. But I was gonna ask if you were a ham because it sounds like, with that kind of going around and a lot of those people are into ham. I started the first ISP in Philadelphia, and a lot of my customers were hams that got into building ISPs because they just love to communicate and stuff like that.

(Joel Beasley at 00:39:20)
Yes. I don't do it regularly, but I have the equipment.

(Avi Freedman at 00:39:24)
Okay.

(Joel Beasley at 00:39:27)
I live an hour outside of Nashville in a farming community. And so we're like the smallest—we have seven acres. We're like the smallest plot. Everyone around us is like 90, 100 acres. But they all talk to each other because it's odd if we've got strangers hanging around. So you don't just come—we're like a secondary road. Like, there's a reason why you're here if you're here or you're lost.

(Avi Freedman at 00:39:54)
"Hey, Ma. Where is this? Confused again?"

(Joel Beasley at 00:39:59)
Right? Actually, when I bought this place, I had to train Google Maps multiple times for it to even direct me back to the correct spot because it thought a road existed that didn't exist. And it was fun.

(Avi Freedman at 00:40:14)
That's why it's important to have a bullshit detector so you don't drive off a cliff because Google told you to.

(Joel Beasley at 00:40:19)
That's the generation under me that has that problem. I got lucky. I was—I tell people I have a brother who's three years older than me. He's a doctor. His sister's three years younger. She's a science teacher. And I was the last generation that was what I call home by streetlights. Right? Because I could see my sister, three years younger than me, and her friends, where it was like you go to jail if you're allowed outside. And then my brother and me were locked outside so mom can watch her soaps and drink from the hose—you're not getting in the house to get Kool-Aid. She needs some me time. And so we were that last generation where they would instruct us to go play in the woods a mile away. So I did get that.

One thing I really wanted to talk with you about, I don't know if you study it a lot, but I know you see a lot of traffic, and you know a lot about the underlying parts of the internet and the different layers. I was having a conversation with a buddy of mine who is Dave McCall, great guy. He's the evangelist at QTS Data Centers. They're like the largest data center by square foot.

(Avi Freedman at 00:41:17)
You know them. Okay. Yep.

(Joel Beasley at 00:41:17)
And Dave's awesome guy. He's got a podcast and everything. And we were having dinner, and they were telling me about how the growth is exploding. So he showed me this graph in his office, and he was like, here's this graph. And he goes, "Point to where GPT came out." Where generative AI became mainstream, you could see it as like January, February of last year.

(Avi Freedman at 00:41:49)
It was like, "Whoa."

(Joel Beasley at 00:41:49)
Yeah. So you saw the hockey stick in data storage compute centers.

(Avi Freedman at 00:41:55)
Have you ever been to Ashburn, Virginia?

(Joel Beasley at 00:42:00)
No.

(Avi Freedman at 00:42:00)
Okay. So the center of the internet—well, it used to be Tyson's Corner. It used to be McLean slash Vienna, Virginia. And then there just wasn't enough space. Then they didn't have road zippers, and there wasn't enough fiber and business-zoned space and power. So they moved out to Ashburn, which is where Dulles is. So the Udvar-Hazy Air and Space Museum expansion with the space shuttle, all that stuff, but Dulles Airport. So around there, just north of there, there used to be—you had some stuff, WorldCom, Equinix had some stuff. And I remember when the first wholesale data center went in because right next to the one where Kentik has some stuff, I had some stuff, my personal colo is—and this massive building, it was DuPont Fabros, which is, I think, now owned by DRT, Digital Realty.

And I was talking to all my friends. And Apple's like, "Oh, yeah. That's our data center." And Microsoft's like, "Yeah. That's our data center." And Facebook's like, "Yeah. That's our data center." There was a beginning of this wholesale where you buy 10 megawatts at a time. Each room is a megawatt and you're buying 10 megawatts, and that was low power density. Because now they're talking about 10, 20, 30 kilowatts a cabinet and water cooling and all that. And after COVID restrictions lifted, every time I've been out there—I know all the back roads to get up to Route 7, all the different areas. And it's like, there's another goddamn multi-100 megawatt data center. Like, whereas—and they've actually, they can do the power. They've just said like they don't want to, which is weird because they have a property tax on equipment in Loudoun County, but I don't really understand that. But, yeah, the demand is massive.

Now what's interesting is you said traffic. So until we'll say ChatGPT, even though there have been these decentralized storage things and people do a lot of storage traffic and megawatts, it's pretty correlated. A lot of the AI stuff does a lot of networking using some, I don't want to say ancient, but older technology called InfiniBand or its derivatives, like Ethernet but isn't Ethernet, and super high bandwidth inside, but doesn't actually drive nearly as much bandwidth as a web server for—or some of this—or a cloud compute infrastructure. So I hope everyone's gonna be okay. Some of my friends in the data center industry have said they worry that some of the smaller competitors are signing long-term commitments for people that think they're going to be around a long time for AI, but might run out of financing, might not get the GPUs. You know? On the other hand, I'm bullish enough that, you know, historically, it'd be a mistake to bet against data centers. We've got 20 massive, you know, call them data center companies, a couple dozen cloud data center and some massive cloud companies as customers, and I wouldn't bet against them. You know, they're just gonna grow over time. But that hockey stick is more internet's growing, but the space demand due to AI and power—power is really what we're talking about with data center. It's not actually about space. The power demand has grown much faster than last year than the bandwidth.

(Joel Beasley at 00:45:40)
Have you seen the people that are building the mobile data centers, mobile compute, essentially, and they're dropping them where there's excess energy, like abandoned solar farms and things like that?

(Avi Freedman at 00:45:52)
I think that makes a lot more sense than some of what people have been doing with edge compute. Like, the idea that every app needs to sit next to every cell tower doesn't really need to be that distributed. Decentralized. Distributed is an architecture thing. Decentralized is a political thing almost. But they're related, but not the same. But, uh, yes. I mean, that was another argument for investing in sort of edge AI was not just that you could do model iteration sort of at the edge, but that you could plug them into the lamp posts and do SETI anywhere with your AI application using power that's gonna be used anyway. I don't know that much about how those folks are doing. I've just read that it's going on.

(Joel Beasley at 00:46:40)
Oh, yeah. I actually just invested in a fund that's doing it.

(Avi Freedman at 00:46:43)
Oh, cool.

(Joel Beasley at 00:46:43)
And so I learned all about it because there's the energy people that know energy. There's the capital people that know capital, and then there's the compute, like, the operators that know.

(Avi Freedman at 00:46:53)
Yep.

(Joel Beasley at 00:46:53)
And so this firm was bringing these three things together and—because the capital doesn't know how to evaluate the deals. They're like it's a compute box that they're putting on this energy, surplus energy.

(Avi Freedman at 00:47:04)
It turns out there's a lot of energy that just needs to be produced. I mean, the way that energy is produced is you can't just turn it off when there's no demand. Like, you have to be producing—otherwise, you pay. It's curtailment cost. Nuclear.

(Joel Beasley at 00:47:14)
You pay.

(Avi Freedman at 00:47:17)
Right. Yeah. Exactly. It's like the cars. You know? It's like if your regenerative brakes, you know, if your battery's too full, you don't have big heat sinks on the back of the car, you can't use the regenerative brakes if the battery's too full. You know, it could be hard to wrap your head around.

(Joel Beasley at 00:47:34)
Yep.

(Avi Freedman at 00:47:35)
For a software guy. Studied physics, and I have built more crappy electronics projects than you'd ever wanna see and discovered that you might need to everywhere.

(Joel Beasley at 00:47:45)
Yeah.

(Avi Freedman at 00:47:47)
Yeah. Exactly. I mean, I have soldered. I have repaired things with solder, but breadboards is my thing. You know? And if I'm gonna hook it up to an actual computer that I care about, like relays and optoisolators and shit like that. So, yes. I used to inhabit—never worked at Radio Shack, but I used to inhabit them for a bit. So...

(Joel Beasley at 00:48:07)
I used to spend quite a bit of time at the—and my dad had like a mini Radio Shack on his workbench with all the little...

(Avi Freedman at 00:48:13)
Well, we had Heathkit and Raider Bend, and my grandfather built machines for people until he built a machine to make his own manufacturing business. So he had a whole thing. We took like reading pacers that we'd go—like, a thing up like this and built like elbow joints, built a robot. When we say we, I mean, my grandfather, but they designed it. Made me do the circuit diagrams, but he was the system architect.

(Avi Freedman at 00:48:34) And, you know, it was a tool to get me to train in safety and using the band saw. And he was like, "No, you need to wear shoes. I cannot wear shoes because I've been working in a machine shop with metal shavings in my socks for fifty years." But, you know, how to—been punished for not putting the tools back, all that stuff.

(Joel Beasley at 00:48:53) So just so I can put a pin on our last thought and wrap that up. You have seen, as far as energy and all that, you have seen—yeah, there is an increase in traffic, but it's not directly one-to-one of the increase in storage and compute because there's a lot of stuff happening internally.

(Avi Freedman at 00:49:09) Storage, it's a little bit closer, but the compute—and the whole separate topic is there's stuff that is really confusing even to people that are really plugged in, which is, are we going to have network meshes or service meshes? Where is all this stuff going to be done? And how is this going to evolve? And are we going to have sidecars or not sidecars?

(Avi Freedman at 00:49:31) And where does load balancing and telemetry and health checking and all this stuff get done? That's confusing. There's a lot of people trying to say, "Well, the storage needs network processors, and the network processors need to know storage, and we're going to have different buses." And there's a lot of that evolving technology, most of which is not mainstream. But generally, when people put up a whole ton of hard disks, it is going to be some bandwidth use, not as much as for the compute. GPU compute often pulls much more locally than sort of over the network.

(Avi Freedman at 00:50:06) So yes, inside that infrastructure, which could be campus-wide, the network's pretty big. But from the internet, which is where we've got 500 data sources—we're seeing about 250 of them let us do aggregate use for Doug Madory's work. That doesn't—that sees inside data centers, but people don't let us aggregate that. They let us aggregate across views of the internet, and the GPU stuff just isn't driving that much internet traffic, even the generative. Yeah.

(Avi Freedman at 00:50:36) There's traffic for some of the generative video and images and videos coming, but it isn't that much. No one could argue it's more than YouTube uploads and stuff like that. So it'd be sort of hidden in other things if the AI is enshittifying us, as Cory Doctorow believes. But I'm not saying he's wrong. I don't think it's—I think it's still noise compared to baseball games that got uploaded to YouTube, but I don't have data to point to for that. You'd have to talk to someone at Google.

(Joel Beasley at 00:51:08) Yeah. I just had a conversation with somebody about Spotify and them working on their issues with storage because people are uploading tons of generated music rapidly. And so now they're trying to put a ceiling on it or a floor, whatever you want to call it. But basically, if your content's not getting X amount of engagement within Y time, then it's not gonna store it, you know.

(Avi Freedman at 00:51:33) Interesting. Yeah.

(Joel Beasley at 00:51:35) Yeah, I'm kind of offloading because they were full. My brother-in-law is a music producer, so he's not a technology guy, but he was—

(Avi Freedman at 00:51:43) Yeah.

(Joel Beasley at 00:51:43) I was at Spotify's headquarters, and they're telling me how their data centers are full and they can't store anything more. I was like, "Well, that's not exactly how storage works." But what they're saying is according to their business model of how much they're gonna spend on storage, they're full.

(Avi Freedman at 00:51:56) They've got to figure out how to do it. Yes.

(Joel Beasley at 00:51:58) Yeah.

(Avi Freedman at 00:51:58) You guys are not allowed—folks, they could do it if they wanted to.

(Joel Beasley at 00:52:03) Oh, they could do it. Yeah. I think that—

(Avi Freedman at 00:52:03) I think enshittification is a much bigger threat than Skynet. You know, if you're gonna be worried about something—

(Joel Beasley at 00:52:09) Can you give me the 30—I don't know this—the 30-second overview of what is this?

(Avi Freedman at 00:52:14) Oh, which? Skynet or enshittification?

(Joel Beasley at 00:52:16) Enshittification.

(Avi Freedman at 00:52:19) So Cory Doctorow, who I've been following since he wrote "When Sysadmins Ruled the Earth," which is a great short story about how Usenet saves the world—basically, a bunch of people in data centers with air filtration are the only people that survive, and then how do they try to reconstruct society or not? He's been writing about the potential dystopian side effects of technology for some time. And he's very concerned about large centralized technology companies and their ability to apply policy with intended and unintended consequences. And also about some of the consequences of AI. And, you know, if you can generate content that looks close, can program people, you know, all this stuff.

(Avi Freedman at 00:53:14) So he's a good guy to follow. I think he's—is he a part owner of Boing Boing? Might be. I don't know.

(Joel Beasley at 00:53:22) What's his last name? Cory what?

(Avi Freedman at 00:53:25) Doctorow. D-O-C-T-O-R-O-W, I think. Probably T-O—

(Joel Beasley at 00:53:30) Google it.

(Avi Freedman at 00:53:31) R-O-W. Sorry, it's late for me. Yeah.

(Joel Beasley at 00:53:33) Yeah. I'm sorry, man. We're we're gonna wrap up here.

(Avi Freedman at 00:53:35) He's a famous science fiction author. Science fiction, fantasy, you know, et cetera. More science fiction, actually. Science fiction.

(Joel Beasley at 00:53:43) Crapound.com looks like his—

(Avi Freedman at 00:53:45) Yeah.

(Joel Beasley at 00:53:45) Yeah. You should spell—

(Avi Freedman at 00:53:48) If you want to follow stuff like this, and Bruce Schneier—sort of, sometimes, if you want to think about—by the way, I didn't coin "privacy versus convenience." I'm sure it was Bruce Schneier or someone like that that did. And so—

(Joel Beasley at 00:53:59) Spell his name again. Bruce?

(Avi Freedman at 00:54:02) S-C-H-N-E-I-E-R, probably. You might not have to—talk. Security.

(Joel Beasley at 00:54:07) But I am a huge fan of movies, and I love—when you said that storyline just now, my mouth was watering. I was like, "Are you kidding me?" That is so unique. I've never seen a movie on that. I've never read a book on that.

(Joel Beasley at 00:54:20) If there's a book about that—it makes so much sense, right? Some virus or something happens, and then they have filtered air.

(Avi Freedman at 00:54:26) They did air filtration, and then they have to do Usenet.

(Joel Beasley at 00:54:30) That's amazing.

(Avi Freedman at 00:54:32) Yeah. Ender's Game filtered out the part that I think is really fascinating about the books, which is they actually posited this worldwide thing called the net where reasoned intellectual discourse happened. They got that wrong. But it was interesting. Yeah.

(Joel Beasley at 00:54:49) That's funny.

(Avi Freedman at 00:54:50) Yeah. There was information warfare because these two brilliant children actually adopted alter egos, Locke and Demosthenes, and sort of did the disinformation thing and programmed the populace toward politics. It was a little prescient.

(Joel Beasley at 00:55:04) These are fascinating assumptions. When you come across that, think of me. Be like—if you come across cool stuff like that, just be like, "I'm just gonna forward this to Joel."

(Avi Freedman at 00:55:11) Yeah. I met him at the World Science Fiction Convention in Chicago in 1991. So I've been following this stuff for a while. Yes.

(Joel Beasley at 00:55:21) Okay.

(Avi Freedman at 00:55:21) If I really don't like you, I'll start forwarding you some LitRPG series that I like, which is a trashy subgenre of both fantasy and science fiction that actually often posits—sometimes the god or goddess picks you up and puts you in a universe where you've got game-type mechanics, and sometimes it's you upload yourself to VR, and sometimes—anyway, it's a—

(Joel Beasley at 00:55:41) Filthy question. Interesting.

(Avi Freedman at 00:55:42) We need to—

(Joel Beasley at 00:55:44) We need to forward some of that stuff to Hollywood and get some of those—all we've got is a couple Ready Player Ones out here.

(Avi Freedman at 00:55:51) Yeah. Yeah.

(Joel Beasley at 00:55:52) Exactly. Yeah. I want to do a little bit of work with you real quick before we wrap up. The problem that people are experiencing when they think that Kentik might be a solution—or if they're having this problem and you see, "Okay, you know, a lot of my customers are having these sets of problems, and that's why they seek us out." For anyone who's listening that might be having these problems, what are they?

(Avi Freedman at 00:56:12) Performance or security problems in your network infrastructure, or you are the people that run that infrastructure, you spend way too much time on bullshit. People reporting problems, and you're still debugging it by the time the problem's going away. You need to work with your peers. And I don't know if you've heard of MTTI—mean time to innocence. So we take all the data, keep it, using—we've used machine learning; we're using LLMs with the appropriate guardrails to try to help people do their jobs better, surface the insights, point them to where they need to look at.

(Avi Freedman at 00:56:46) A little bit of automated remediation. The network industry promises that's possible. I think it's a little bit early for Skynet-driven networks, but making the job of the network professionals easier, helping them work with other groups—sometimes that's security. Usually, it's more operations. And, you know, on the business side, if your business is dependent on the network to connect your people, to have your revenue flow—the other observability systems don't really understand the network infrastructure unless you're only super hipster eBPF in the cloud, which is—Kubernetes doesn't have an eBPF driver.

(Avi Freedman at 00:57:27) So it still needs to generate packets that flow over some network somewhere, and someone's gotta make it work. So that's Kentik's jam.

(Joel Beasley at 00:57:35) Nailed it. It's just kentik.com, the network observability platform. "Everything you ever wanted to know about your network from data center to container to cloud." I'm just reading off your home page. It's very elegantly spoken.

(Joel Beasley at 00:57:49) Thank you. Avi, thank you so much for doing this. We made a podcast. How do you feel?

(Avi Freedman at 00:57:55) Awesome.

(Joel Beasley at 00:57:55) Thank you so much for listening. And if you found this episode useful, please share it with a friend or colleague who you think would get value from it. And if you have topics that you'd like to hear discussed on the podcast, either add me on LinkedIn or send me an email, [email protected]. Every time I get an email or LinkedIn message, it absolutely makes my day and inspires me to keep going.