Thomas Hazel - CTO at ChaosSearch

(Joel Beasley at 00:00:00) Hello, my friends. Today we are talking to Thomas, the founder and CTO at ChaosSearch, and we discuss their mind-blowing innovation in data storage and search, how constraints can breed creativity, and why data is cheap but information is expensive. All of this right here, right now on the Modern CTO podcast.

(Thomas at 00:00:21) Here we go.

(Joel Beasley at 00:00:22) This is the Modern CTO podcast. I'm going to tell you the moment that I fell in love with you or the company. I was reading your blog, and you had—there's this line in one of your blog posts that said ChaosSearch, right, was derived from an exercise of first principles. Personally, I am a huge fan of first principles. The moment I heard Elon Musk describe it in some interview I was watching with him, I put it in my phone as a recurring event to come up every three months for the rest of my life because it's just something you always have to go back to.

(Thomas at 00:01:02) I love it. I love it. You know, and to be frank, I'm an inventor. Right? So if you don't follow your first principles, you'll get lost. Right? And so you think about what you want to do, come up with some of those basic constructs, those axioms, and say, "Okay, I think this is right. Let's go for it." And both building the team, the product, you know, the mathematics, right, is always, "Does it fit within that viewpoint?" If it doesn't, it's probably wrong. Right? And to your point, I think Elon obviously is a unique individual where he thinks outside the box. And I think those first principles help him really keep a North Star, right, where he's trying to go. And so, yeah, thank you. I believe in it as well. I'm obviously not the inventor of first principles, but I'm definitely going to be the follower.

(Joel Beasley at 00:01:54) Yeah, I like those concepts. I find that at first, I was collecting a bunch of these concepts when I was getting excited about learning and having more self-awareness in my late twenties. And then I found that there was a lot of value in just taking a handful of them and consistently revisiting them rather than a constant stream of new ones. And so I took that approach, and so I was layering down on the first principles and then a couple other ones. And I'm finding that there's a nice advantage to having a depth of experience with one of these concepts.

(Thomas at 00:02:27) It's so funny you say that because I 100% follow a similar path. I'm a—maybe I'm a little older, but you know, one of my first principles is when I see a problem, I invert it. It's the first thing I ever do. So if everyone's going that way, I just look the other direction and see what I see. It's so basic. And so any new thought or new invention—and when I say new, new to be relative. Right? But everyone's always looking that way. If you look behind you, it's amazing what you see. And so whenever I see a problem, I say, "Okay, they're doing it that way. Those problems have been solved that way. Just turn around. What do you see?" And that's one of those first principles of solving problems that I jump into, and you'd be amazed just saying, "Well, okay, that's fine. But what if I did it just the opposite way?" And I don't want to say 100% of the time, but it works more often than not. It's just such an amazing thing.

(Joel Beasley at 00:03:24) Massively useful tactic. Like, I found—I think we just became best friends because you know how you think a certain way and it's like you've been a certain way for a certain length of time and you're aware, "Okay, I've been like this for this amount of time." And then you meet someone, and I rarely meet other people. I don't think I've ever met anyone else who's described that to me, and I do that constantly. I don't know why I do it, but I just—maybe I had success with it the first couple times. But I always say problems are just solutions that haven't been converted yet, which is, you know, similar to, like, look at the inverse.

(Thomas at 00:03:58) Well, you know, it's funny. I'm a computer science nerd. I'm a mathematical nerd, I guess. I love information theory. And, you know, there's a thing in art where they try to change their perspective just to see the information differently. And I can't remember what the technique is, but that essence is what you're trying to do is look at the problem in a different perspective, a different light, and hopefully insights or something, the awakening of going, "Ah, I see something here that I didn't see before." You know, how often do you maybe write a paper and then you can't seem to change it, but you walk away for a day, you reread it, and it comes back to life. I mean, to your point, you practice this. You practice those techniques so that you have a new eye. There's a term, having a new eye on information or on a problem, that as you tweak it, as you turn it, if you can really see a different perspective, those problems get easier. And that's something that I personally—to your point, I didn't know I was doing that when I was younger, but there's little things that I started picking up these techniques that when I got older, I heard people say things like Elon Musk, and you're like, "Wait, that's what I do. Okay." And then hopefully, you teach other people those exact same things. You know? I love to write one of those, like, you know, "For Dummies" books. Right? And how to have a new eye on solving problems because, you know, the idea that with ChaosSearch—and we can get into it—was one of those exercises where, you know, having years of building big distributed systems coming from telecom to distributed systems to ultimately information and information at scale or big data. Why can't we scale more? Why are these so much pain and cost and complexity? And the easy thing I did says, "Let's throw it all away." You know, it sounds so simple, right, to say that, but you'd be amazed, or at least I'm amazed, how often people stick with what they know and try to patch, try to rework. You know, a lot of the database technology that's out there is thirty years old. Now we repurpose it. We move it around. Maybe we have faster computers, faster networks, but it's still the same technology. And so at Chaos, I said, "I know the limits. Now let's change it. Let's try to rethink those limits or just throw it all away." And so I really appreciate that feedback because I hope this for all people. Right? Hopefully, someone listens to this podcast and says, "You know what? Let me try it. Let me try that technique."

(Joel Beasley at 00:06:34) I think everyone's probably really interested to know what ChaosSearch is now. They were talking about it. They're like, "We love the first principles. Like, what is ChaosSearch?"

(Thomas at 00:06:43) So ChaosSearch at a high level, if folks out there know big data or cloud or data analytics, information is now driver force, the fuel that drives the information economy. Data is getting bigger. Problems are being solved with existing technology. So with Chaos, I said, "I have an idea. What if I change how information is stored? What if I change where information is stored? What if I change how information is searched? What if I change how information is compressed, means shrunk?" And the idea behind ChaosSearch was a new invention, a new technology that I created—really a file format that represents information that can uniquely be leveraged in this term called cloud object storage where all this data is being put in today. You know, so Amazon has their first service, Amazon—you know, we think of Amazon, they go book company. Right? Well, now it's a cloud company, and their first service was cloud storage. The awakening that they had of saying, "I know we're a book company. I know that we have really good scaling back store. Why don't we do this as a service and make it offering to the company or to companies?" And so the idea behind ChaosSearch was transform that cloud storage into a new type of database using this unique technology and opening up access to information. So business operations, business intelligence, all at great scale becomes costly, complicated. And with the Chaos service, we connect to your cloud storage and unlock those analytics.

(Joel Beasley at 00:08:26) Let's bring it down a little bit closer to home. Like, I read you had a HubSpot case study. You see you guys worked with those people over there. Great people, by the way. I know some of the original people that have since left and gone on to do other things like Elias Torres built Drift after leaving HubSpot. But, um, what did you do? Like, what's the value? Like, how did you help them?

(Thomas at 00:08:50) Oh, great. So they are a success story right here in Boston. They had unique perspective on how they want to deliver marketing and websites to customers, and their company is run by information. Customers come into the solution, and behind the scenes, logs are generated. And those logs are the operational lifeblood of how that business runs. And they were running this software stack called ELK or Elastic Stack. And ELK is a tool or a service that allows them to debug what's going on in their company from an operational perspective. So for instance, they offer websites for companies to host their domains. And as a denial-of-service attack comes in, they need to know what's going on. And they use the ELK stack to figure out, "Hey, what IP address is causing a problem to ultimately turn that IP address off?" and other things that they do in operations. And so they were running the ELK stack manually because of the cost-prohibitive nature of using some more expensive hosted solution. But when they say cost-effective, you're spending millions of dollars for one use case on logs. You know there's a problem, and the data is getting bigger. So last year, for this one use case, this Cloudflare data, the CDN dataset, every year, every six months, it was doing another five terabytes of growth per day. So it started out when we first started talking to them at five terabytes a day, then it went to 10. Now it's at 15 and growing. So it's growing because they're a success. Those customers that they're offering solution to are growing. Great. However, when your one use case of log analysis is millions, then becomes two million, et cetera, et cetera, et cetera, you see that there's a wall that it becomes unattainable. So what happens, companies shrink the amount of data they can analyze to keep those costs down. And so they reached out to us. They said, "Hey, I hear that ChaosSearch transforms—because they're already using object storage S3 for all their log storage—transform their storage into an ELK-like solution to provide log analytics." And that's exactly what we did. We came in. They ran our service on their S3 storage and did the exact same use case of denial-of-service attack, for instance. And instead of spending, you know, millions of dollars per year, now it's in the hundreds of thousands. So, you know, that's a dramatic cost saving, but a couple things that happened. Now they don't have full-time engineers trying to keep the ELK stack running. Not only is it expensive to run at scale, but it falls over a lot. It's a known problem. And the other aspect is we can grow the retention. So now they just don't store, you know, five days of retention. They can do two weeks a month, ninety days. And the ability to know what's going on, not just immediately, but over time is a key aspect that they need in their service. So and they're adding more workloads. You know, first, it's Cloudflare, then it's NGINX. You know, you can imagine with our service, you enable a data lake philosophy because we've merged cloud storage with analytics. So you just stream your data into cloud storage, and our service takes it up and provides analytical APIs via the ELK Stack-compliant tooling.

(Joel Beasley at 00:12:26) That is exciting. That is—you know, developers and logs, they just grow exponentially. And the fact that you can extract meaningful insights, it's more than just, like, denial-of-service. You can extract some actual, like, meaningful information from these data lakes. Right?

(Thomas at 00:12:42) Absolutely. And so that's the thing is there's so much information in the company, whether it's operational logs, business intelligence logs, information drives companies. And if you have a slow website or the website is down or you have supply service that's not working, you need to know it immediately, and then you do forensics on that data over time. And data is getting so big. Your machine-generated data is outpacing Moore's law, meaning that it's so easy to generate information. It's so hard to search and find it cost-effectively. And so that's where ChaosSearch comes in, and that's where the inventions behind it has addressed those problems.

(Joel Beasley at 00:13:24) I love it. What inspired the name?

(Thomas at 00:13:27) So you can imagine, you know, big data, entropy, chaos. So what if you could wrestle and search the chaos? And so the idea is that let your data be chaotic. Our service will come in, discover it, catalog it, index it to ultimately search and query. So, you know, it's one of those things where I love information theory. I love chaos theory. I love entropy. I like to, if you will, wrestle the ground, entropy and disorder. And so that's kind of my life's work, to be frank. And so if I can make information small and make it accessible, it really helps out people like HubSpot where think about it. Terabytes is what you have. Right? Think about terabyte. I think IDC said in 2025, there's going to be 175 zettabytes of data being generated. Google zettabyte. And when you see how big that is, information is—when they say this information age now, it's just exponentially growing. And the technology, the science is still thirty years old. You know, from nineteen seventies, these systems were not designed to handle, let alone a petabyte. Maybe a gigabyte was big back then. Right? So you can imagine what a zettabyte would look like.

(Joel Beasley at 00:14:47) I did Google it. It's a million petabytes. Yeah. Yeah. That's insane. The zeros is just unbelievable on the Google search for that. Wow. And so, like, here's a good question for you that I was thinking about earlier today. Okay. So hypothetically. Right? Or let's—it doesn't have to be hypothetically. Did you see during the pandemic that the Pentagon released the UFO videos officially? That's very exciting for me. Regardless of if they're aliens or not, let's like, they're still unidentified flying objects that are that they're observing. Let's pretend that they are aliens. Right? And they came to Earth, and then what they do in the atmosphere or whatever, they look at—they use our data, like, the volume of data we transfer as a way to understand where our maturity is as a civilization.

(Thomas at 00:15:41) Yeah.

(Joel Beasley at 00:15:41) Right? Just on that experiment. Right? What do we—like, what do you think the most data transfer is? Like, how would you go about—like, let's just explore it off the top of your head. Like, how would you go about figuring out a pie chart of the volume of data transfer and what type of data it is that we generate?

(Thomas at 00:16:00) Question. There's been a whole bunch of statistics. Right? And they're statistics. They're not just that. Right? They're statistics. You know, I come from telecom back in the day where I used to build these big, mammoth, gargantuan boxes that moved lots of data. So I've seen the pipes that get moved. Now the type of data, you know, there's some unique things online where is it video? Is it—it used to be telephone. Right? Usually telephone information. But now it's machine-generated information where all these devices, IoT, it just—there's so much content being generated, and there's a value in that data. So when the aliens look at us, they go, "That's an interesting app. Why is TikTok so popular?" You know? And they're—I think they're judging us for that just alone with the video and time where they spent a lot of money and time on that. So maybe we're not ready to take it to the next level. But, you know, it's human—it's human experience. We're driving the need to develop those applications, those devices.

(Thomas at 00:17:05) So I would say the majority of data is new data that is app device driven. But then again, you have some old school scenarios where if you're trying to crack an atom within a second, you're generating 100 terabytes. So I would say the data, it's always been generated. Now we can capture it, back to Moore's Law.

(Thomas at 00:17:32) With Moore's Law now, we can store more data than we ever could before. Right? And so now we think we can utilize that data today. And that's where ChaosSearch comes in. Okay, we know we can generate it.

(Thomas at 00:17:46) We know we can store it. Now how do we access it? And that's where we come in.

(Joel Beasley at 00:17:51) I'm gonna improve that question over time. That was actually, I just thought about it for the first time, like, last night as I was falling asleep, and I wrote it down real quick in my Evernote, and then I woke up this morning and refined it a little bit. But I wanna ask a good question around that. I wanna take your response and then improve the question further and let that evolve because I just think it's a fun one.

(Thomas at 00:18:15) Well, I mean, information is power. We've heard this. Knowledge is power. The question is, what's the problem? Is it the inability to generate? I like to say this is another thing that I've written about: data's cheap, information's expensive.

(Thomas at 00:18:34) What I mean by that, it's so easy to generate data. It's now relatively easy to store data. But data is just that. It has no context. It has no value.

(Thomas at 00:18:46) When you create meaning out of data, it becomes information, and information can be utilized. Let's say COVID. Right? If we had all the information of how it was spreading as it was spreading in real time, we would know a lot more about what we should have done prior to what we decided to do as we saw more of a reflection of what we should do.

(Thomas at 00:19:09) So for me, the ability to analyze data at scale really changes all our lives. And I'm assuming those aliens have mastered information. Right? Whether it's time travel or, you know, because if you ask Hawking, right? Black holes and entropy and information, it's all the same. Energy, information is one and the same. I think we master information, we master energy, solve world hunger and the energy problem, I hope.

(Joel Beasley at 00:19:40) I'm looking forward to it. And that was actually one of the, I can never remember the name. It was like the Kardashev Scale, or the K Scale is what I say in my head. But they measure maturity of civilization based off energy consumption or manipulation.

(Joel Beasley at 00:19:55) And I was wondering if there's a more micro way, because he has giant leaps, and I was wondering if there's a more micro way. Because I think it'd be interesting to see a graph of, like, if we saw a graph of humanity's attention and where it's placed, I think that'd be a really interesting visualization.

(Thomas at 00:20:14) No, you're absolutely right. And I do believe what you were saying, where I do believe energy and information have a relationship, and I would argue if you solve either one of those, you solve the other. I'm always worried about what, if I could see everything, what I would see all of us doing with all this power of information where, you know, half the time people are playing on the games or doing TikTok versus, for me, solving really unique hard problems. I'd hope that it's 50/50 at least of just fun versus value.

(Thomas at 00:20:51) But, you know, I think sometimes it's just, I'm at the foot of the aliens are saying, oh, they're not ready. They're not ready.

(Joel Beasley at 00:20:58) They need to get the time to cook. Right? I mean, I think that's a reasonable assumption too. Like, they would just come down and they'd see everything at once and just know whether or not we're at a point where we're ready to interact. One thing I am curious about, quantum computing, and specifically because I have been researching it quite heavily over the past couple weeks.

(Joel Beasley at 00:21:27) And you're very smart, you're math. And one of the first quantum algorithms ever made was for search, like a very basic, you could, that's the closest analogy you could say to it. It's not like a full blown search. Now have you looked at quantum computing at all?

(Thomas at 00:21:45) So I have. I would like to say if I wasn't a pure scientist, I'd be a physics major. At the time, I didn't know how to make a career out of that. So I love, you know, whether it's quantum mechanics or quantum computing. You know, what's interesting to me is how does it manifest into using it? Right? Does it act like a computer today? Do I have to have higher expertise in utilizing the parallelism or the multidimensionality of an algorithm? Right? Coding can be hard.

(Thomas at 00:22:19) When it becomes distributed, it becomes even harder. When it becomes multidimensional, it becomes yet even harder. So there's gonna be some great languages that will have to come out as really intelligent people to make quantum computing as easy as, let's call it Python, which is a popular language these days. I'm excited for it because you can imagine the ability to have that type of analytical execution at a fingertip. The fear is, right, all these algorithms for security can be broken instantaneously because you can crack a code because you can put that much compute into an exercise.

(Thomas at 00:22:58) So I'm excited about what one would build with that type of execution power. I always, this goes back to the inverter thinking. If you could, what would you see? And so, you know, I haven't put any thought into it, although you mentioned that you've been studying it. I haven't put any thought to what would I build if I could?

(Thomas at 00:23:20) What would we do? I'm not sure, but now that I'm intrigued, I might just go spend a few weekends thinking it through.

(Joel Beasley at 00:23:28) I'll give you my, like, 30 second overview of what I found. And my background is software engineering. And so I was looking for it to be at that level of, like, let's do business things and logic and speak to it in English and not machine language. It's early. I'd say it's analogous to the computers being the size of a room, right?

(Joel Beasley at 00:23:50) Where they're doing some very basic things, but yet there's still a quantum API you could plug into today and run some quantum code on. I actually did this basic 10 minute tutorial with, like, a six sided dice roller that was actually processed on a quantum API. But it seems like it's really close. Like, it's close to what? Right?

(Joel Beasley at 00:24:15) To what point in time? But if you're there, it's evolving. There's money there. It is a market. There are people that are writing APIs for Python and making it more accessible every day. You know, the top companies have all been doubling the number of qubits they could have and competing against that. You know, there's a press release from Amazon, then Microsoft, then IBM. And then out of nowhere, a couple weeks ago, we saw one from Honeywell. And they're like, yeah.

(Joel Beasley at 00:24:47) We just created the fastest quantum computing system. And so I actually got to talk to the head of that project, Tony, and, like, get really detailed in on where the state of things are. And I would say, like, after all of the content I've consumed, including courses on linear algebra to better understand quantum mechanics, of everything I consumed, I would say it is worth taking a person or two that you trust that's intelligent and having them do some quick project of, like, hey, put together a timeline of when quantum computing will be relevant to us and when we could start playing with it. Because I think the people who play with it now will still be, like, right on the edge early.

(Joel Beasley at 00:25:31) Like, they're not exactly early, but they're right on the edge. I think now is a good time for people to just explore.

(Thomas at 00:25:36) You know, it's funny. Like, so I work on algorithm or I think about information all the time. And if I had that type of compute, what would I do differently? Right? These are things that I think about. Where could you really crack random? Right? Could you crack, you know, distributed hashes? Could you crack all those things where, you know, maybe you come up with a different data representation for databases, right, from chaos?

(Thomas at 00:26:08) Because, you know, in the end, whatever you choose to do has a time vector. Right? So I like to say, if I can make information zero, I make time zero. But if I can make time zero, I make information zero. And so with quantum computing, you're getting time close to zero. Wow.

(Thomas at 00:26:22) Right? Think about what you could do with information when it becomes that quick.

(Joel Beasley at 00:26:28) It's beautiful. It's brilliant. I love it. I'm so fascinated by it. I'm grateful to be alive, like, during this time because, you know, people who are alive maybe during the computer time, maybe they didn't have an analogy to know exactly what the possibility of the computer systems were.

(Joel Beasley at 00:26:46) But now we can see it's the future and jump in there and be a part of it. And I'm pretty excited about it.

(Thomas at 00:26:51) Oh, no. Absolutely. Like I said, I have about 10 important problems I wanna solve if I had access to a computer like that. I'm sure I was, I typically choose Saturday mornings, fully awake, fully caffeinated, and then let my mind wander. So if I had access to one of those, maybe I'll have a little bit of fun.

(Joel Beasley at 00:27:13) Yeah. My problem number one on that list would be not having infinite money. Quantum computer solves.

(Thomas at 00:27:21) Yeah. Yeah. Yeah. No. That's good.

(Thomas at 00:27:23) That's good.

(Joel Beasley at 00:27:23) So you got inducted into the Hall of Fame for University of New Hampshire. How, I think it's fantastic, but I'm really curious. Like, what was it actually like? They call you up one day, hey, we're inducting you into the hall of fame. Like, how did that happen?

(Thomas at 00:27:38) Well, so it was a surprise, to be frank. Right? So you go to school because you love or you have a passion for something, and I had a passion for computer science, for mathematics, solving those type of problems. I never thought I'd be a startup person, the person that invents things to build companies, to ultimately sell companies to do it again. Never to be an entrepreneur. I don't even understand that.

(Thomas at 00:28:01) And now people take classes on it. Right? How to be an entrepreneur, how to be a jerk. So for me, I just wanted to have fun and invent and create. But the problem was I did it a few times successfully. And UNH has this hall of fame for entrepreneurs, and they called me up.

(Thomas at 00:28:22) They said, good news. We've nominated you to be one of the candidates. What do you think? I'm like, cool. And then I won.

(Thomas at 00:28:36) And so it's kinda surreal, and I'm proud of it, to be frank, because, you know, like any university, to be nominated to be a Hall of Fame is something to be humbled about. But it was all accidental, to be real frank. I never set out like, oh, I wanna be an entrepreneur or solve these type of problems. I just love it. You know?

(Thomas at 00:28:59) I won't tell my investors or anybody, but I do it for free. I would do it for free. You know? And I think if you love it, that's why you're successful at it. And if you spend enough time, you know, like, was it The Outliers, you know, 10,000 hours?

(Thomas at 00:29:15) I believe in that term. Just commit to it, and good things come out of it.

(Joel Beasley at 00:29:22) I love that too because I read a lot of those books, The Outliers and Grit and all of these things and wanting to understand the find your passion type deal. And what I realized, the conclusion I came to after consuming all of that content was, you find something, you stick with it, you gain some skill at it, you persist through the difficult moments, and you just keep building and building and building, and you just don't quit. And then you'll develop this relationship with the topic or the skill, and it'll just become more and more a part of you. And, you know, everything's difficult. It's amazing too because I can back up and see some people I know, and they'll just bounce from one thing to the next every, you know, two or three months, like a new interest, and they'll have no depth on any of them.

(Joel Beasley at 00:30:13) And they're always just like, well, yeah. It's hard. Or I couldn't make money there, like, whatever it was. And I'm still looking for my passion. It wasn't what I really wanna do.

(Joel Beasley at 00:30:21) I lost passion after three months. Uh, hello. That's how it goes.

(Thomas at 00:30:27) Yeah. I know. It's funny. I mean, there's certain people that are built for this, and I would say I didn't know I was built for this, but I was, where information, learning is such a joyful thing. You know that Matrix, you put that thing in your head and you just imply all that knowledge in your head.

(Thomas at 00:30:43) I would love something like that. That'd be fantastic. And the journey is being part of this discovery, this joy, and finding people along the journey with you that had the exact same passion. You know, so I've been successful, but I've been with a team of people that, hopefully, I've inspired to join this journey. And I've met people who just have that love, that passion to do anything, like Elon Musk.

(Thomas at 00:31:09) Think about it. He did PayPal. Right? And then he did, you know, SpaceX. You think that they're so far apart, but they're really not.

(Thomas at 00:31:19) They're just, he had an idea, and he drove to that idea. I believe in this term called momentum. It's not about your product ideas. It's not about the market opportunity. It's about momentum.

(Thomas at 00:31:31) And I think if you've had that joy, that love to consume and try and push, you create momentum, and momentum moves more, adds more people. And, ultimately, at some point in time, something happens.

(Joel Beasley at 00:31:46) Yes. I mean, I am a huge fan. I've found that momentum is incredibly important and some people that are more magical than others because we get to work with a lot of people. Right? So we get to, when you have a, and then the next thing that's important is, I'm talking about right now, volume.

(Joel Beasley at 00:32:02) So, like, if I were to go back, future self, I'd be like, pay attention to, you know, volume. Like, everything you do, if you wanna be successful, you're gonna have to do, like, 100 times more activity than you think. If you wanna make a sale, you're gonna have to contact way more people than you think. But the beautiful thing is you've got numbers. Like, if you do the volume, you will get a result. It will happen.

(Thomas at 00:32:22) All right. So let's say you go to a university, you get a PhD in some computer science something. Right? If you haven't touched it, if you haven't felt information, at least in my case, you're missing those insights. That 10,000 hours gives you experiences, and your unconscious is working on that.

(Thomas at 00:32:40) And so as you go through and try to solve these problems, all that touching and feeling of the problem intimately really gives you that insight. So the 10,000 hours is not just practice. It's that intuitive nature of what you're working on that you can expand. And so if you look at anybody in, say, in science, they just say one day they woke up and they say, I don't know, gravity with Newton. Newton was a crazy mathematical numbers doing charts and working out equations constantly, constantly, constantly.

(Thomas at 00:33:12) Because he just, he wanted to practice what he was trying to discover, and the insight came from that exercise. So I would say any good entrepreneur is spending wonderful amount of time enjoying it, hopefully, while doing it. And, hopefully, that one moment where you go, wait a second. Wait a second. There's something here.

(Thomas at 00:33:35) And that awakening, you know, to me, what I love to do is with ChaosSearch technology, about five, six years ago, I had an awakening. I saw something. And what happened from that awakening was an insight that as I apply this awakening to problems, you're like, there's another way of doing it. There's another way of doing it. And that goes back to those called full circle base principles, those core axioms that now you have an insight that when someone says in your office or on your team, how would I do this? It doesn't make any sense. Use this core principle. It'll get you there. Because if I will, the math should work. And I think ten thousand hours or whatever the number is is key to that success.

(Joel Beasley at 00:34:20) Yeah. It's like if you to expand upon your Elon Musk thing, if you look at him, if you look at it like, okay, you go PayPal, SpaceX, SolarCity, that seems very distributed. But if you look at it like, here is a human with a collection of principles, it's almost similar to music where you have a number of chords, but the song can be vastly different. And so, you know, that's helped shape my perception of new team members, who we want, like, the culture in the organization, things like that. How involved are you with the company? I know you said at the beginning that you mainly see yourself as an inventor. So what's your role like as the CTO today?

(Thomas at 00:35:06) So as a startup person, right, you're everything. You're the janitor. You're the CFO. You know? But as you grow, you get more specialized. I would say I love building companies. I love building teams. So I'm kind of a hybrid where I invent something. I see where I would apply this invention, identify the market, and then start reaching out to my colleagues that I've worked with in the past that I sometimes mentored myself and say, I have an idea. I have a technology. I have a market opportunity. Would you join me in this journey? And over the last twenty-five years, I've collected, for lack of a better term, a whole bunch of people that love working on terribly hard problems. And they always ask me, what's the next startup idea? What are we doing next? And they raise their hand, and they say, let's go. So I think from day to day, I'm part team builder, part energy engineering manager. I still code, although I know I need to grow out of that. But the creation of the algorithm, creation of the idea was where my true joy was. But now we're in the execution phase of a company like ours where you proved out the technology, you built the product, now you're bringing it to the market. And I'm doing things like this, talking to interesting people like yourself, talk about our journey. So it's the next phase of a company, but I can nerd out. I can be in a cubicle sitting there for years on end and just crunching time on it. So I like both sides, and I guess now is the time to reach out and talk to people like yourself.

(Joel Beasley at 00:36:48) Yeah. I can get myself excited about pretty much anything. I can find the positive, I can find the problem, find out how to make it unique and interesting just because I am not wired to sit there and be bored and do nothing. Like, that is the worst thing to me. If I even if I had a basic data entry task, I'd be doing the task and figuring out what's the one small thing I can improve today, and then over the course of the year, it's gonna turn into unbelievable amazingness.

(Thomas at 00:37:18) That's it. Any task you have, make it brilliant. Make it awesome. Rethink it. And if you see, to your point, you see the world that way, where every idea is a challenge, make it better. So whether you are conducting people to cross the street and you're, you know, you're a stop sign holder, be the best one possible.

(Joel Beasley at 00:37:40) Be the viral video person who learns how to manipulate that environment incredibly well.

(Thomas at 00:37:47) And that's the joy. That's the joy of making things better, making things unique and different. And when I've met people, hopefully, like you and myself, they are so happy all the time. You're like, whenever there's a problem, let's say there's an issue or complexity, I'm always like, good. We have something to work on now. Right? There's no problem. And, also, one of my philosophical core principles, there's no failure. Failure was just a learning moment to apply to whatever momentum you're trying to do. Failures, I don't believe in failure because that was just an exercise of knowledge. You know? And there's no problem. It's just an exercise of a data point of knowledge to move whatever you're moving forward because a problem is your definition of what just happened. A failure is your definition of what just, you know, it's subjective. Yeah. It's so subjective. And so you think about us putting people on the moon. They were failures, but were they failures? Were they learning? Was it part of the process? Right? You think about anything that happens, was that supposed to be understood? And if it was supposed to be understood, by definition, it's actually learning. It's not a problem or a failure.

(Joel Beasley at 00:39:04) Preach. I love it.

(Thomas at 00:39:07) You know, because think about it. People at work, we work on really hard stuff, and I hired people out of school, and they've never done hard like this. Everyone I hired, like, I thought I knew hard, but then this is hard. And if you let people have an open space of thought, right, where there's no time pressure relatively speaking, but they're free to think through what you could do, my job is to ask questions, pause, let your unconscious, hopefully, speak. Because I find so often, just like we go to your teacher, you ask them a question about a problem, then as you're explaining it, you actually know the answer. Right? I believe in that principle where the student or the employee actually has the answer. They have the knowledge at ten thousand hours. They have it. How do we pull it out? How do we bring it together so that your creative mind is gonna answer for us? And I think right now, my job at Chaos is taking wonderfully talented people with wonderfully hard problems and working with them to solve them because everything that we do hasn't been done before. We're not using existing computer science technology. We're not doing it the old way. Every time we need to do something, it's gonna be new. And that's the fun. But, you know, if we do it right, there's the value.

(Joel Beasley at 00:40:26) Is that how you were able to get the cost down so much?

(Thomas at 00:40:29) Yeah. So think about it. So I like to have this, I'm actually coming up with a new blog or article called the Multiplicities of Small. And that's where I came to start ChaosSearch was there was a couple of things that were going on with information. Back to if I can make information zero, I might time zero. Right? Well, I can't make time zero. I don't have a quantum computer, so I'm gonna go on the information side. So the idea is that if I could store information smaller than anybody else, I win. I store less. I move less. I process less. So I started with not just I know I wanted to build a new database system. I wanted to know how small I can make information. So I started with compression, meaning that what were the problems with compression? Why do compressions work on this and not on that? Why does it take so much CPU? Why does it, it's quick, but it doesn't reduce enough? So what I went with was let me work on a new way of representing information that can make information as small as possible. And so what I would have to do is we know database technology. You can have, you know, different algorithms, different representations, column store, text store, row store. I ignored all that. I started with information compression. And what happened was an awakening. Right? When I was playing with numbers, right, playing with bits, playing with, I had an awakening of how I could represent information that could be smaller than compression items like gzip, but do it in a way that allowed me to do search and analytics on it. And that was like, my head started going, I had to stop and go, okay. Write this down. Save it away. You know? Because I'm like, if the world comes to an end, someone's gotta hear it. Someone's gotta hear it. And that was about five years ago. And so what I do is I put those on the shelf. Okay. My next company. I'm gonna take that. And so when I was ready to do my next company, I said, what technology have I solved, and how do I apply it? So this new index technology or new representation compression, where would I apply it? And the idea was cloud object storage, where all this data is going. But cloud storage is not a database-backed type storage or, like, block storage. So the idea behind chaos was making information as small as possible and then providing analytical access to that. And unlike existing technologists now 70 years old, there's walls, there's cliffs. So you start what they call partitioning or sharding information, little silos of clusters to try to connect everything on. It adds time. It adds cost. So the idea is really the multiplicity of small, and that's what we focus on. That's what I started at Chaos, but build a whole product around it.

(Joel Beasley at 00:43:14) Oh, that's so exciting. How did, so and the reason why I asked because you do have the graph on your site that shows, like, the competitors, and you could show how much data transfer you have. And so I was very excited to hear about that. So innovation in the compression of data, making it small and easier to search. That's very smart. Very first principles. I love it.

(Thomas at 00:43:32) So it's funny. So that was the first of the first principles, make data small. Then the next principle was how to be a database index. It had to be distributed. It had to have the ability to auto shard, if you will. So there's all these vectors that you say, here's all these things it has to do. And then you look at technology out there, could those do this? And you say, no. No. No. No. No. And so you start playing. To be honest, you start just putting out a pencil and pen and saying, what if I did this? What if I did this? And what happened was when I created this new representation, it came really small. But then when I saw what I did, I was like, wait a second. I can do high performance text search like an inverted index could do as well as relational analytics like a column store or a b-tree technology would do. And so I could do it on one representation and make it wonderfully small. And I'm like, this is good. This is good. I need to do something with this. And, again, having a team of people I've worked with for years that knew how to build distributed systems, I raised my hand. I said, everyone, I think I got it. Let's go for it. And we started, you know, ChaosSearch roughly, you know, four to five years ago. Now, you know, databases take a long time. So we came to market last year, but we've been building this thing maybe even for a lifetime. Right?

(Joel Beasley at 00:44:52) Yeah. Well, it is a culmination of all your previous experience. So, man, I love it. I love the constraints breeding creativity concept that is in there and how you describe it. I really enjoy the style and how you think.

(Thomas at 00:45:09) Awesome. Yeah. And you think about it, you know, it's something that we've talked about when we first started talking is these are things that you can hopefully, people listen to this podcast and go, how could I apply that to my own job? How can I see a problem differently? And the whole point back to look the opposite direction, what do you see? If you're fighting something, don't do it that way. You know? And then it's really that simple. Take a break. Think about it differently. Look at every angle, and the hope is your unconscious gives you a glimmer of thought, and you try that thought out. And once you start pulling that thread, the awakenings start happening. And with ChaosSearch, we have, I have a big ball chaos index that it's still giving gifts to me. When I created it about five years ago, I thought it was gonna do just this. But then I started applying those core principles, and it started doing that and then this and then that. And I was like, holy cow. It's bigger than I even imagined. And that's been such the joy. It's like you talk about music. When you write that song and it's catchy, you're like, oh, there's something here. And it just flows. It flows. If you talk to a musician, within thirty seconds, they have the song. Right? But they're waiting for that moment. Right? They could be a year, could be two years, but when they have that moment, it just comes out on paper, and that's how this technology of chaos was. I felt it. I saw it, and it was just there. And I don't know if that was because the unconscious was ready to go. All the puzzle pieces just came together because I've been thinking about it. But that's the joy, and that's the fun. And you find a group of talented employees that love to do those type of problems, they can inspire. Because as I do a whiteboard exercise of how it works, their minds are going, holy cow. How come we haven't done this before? I don't know. But we are.

(Joel Beasley at 00:47:07) It's like gifts. It's another way I've been thinking about it. It reminds me of my rough understanding of blockchain where you've got these nodes and they're just processing and processing and processing and then they get a reward. They just continually process, process, process and then they get a reward. And then it's like if you, yeah, when I, there was a time in my life when I was not great, and so I made the decision to be great. And when I did that, I figured, okay. Well, I'm not quite sure what to do with these people who I'm studying. I bought all the books that I could find that were life stories of billionaires, like Musk, Bezos, all these people. And I started studying them and understanding, like, how they did great things and all of that because life is short. And what I came up with after reading all of those things was, okay, they all have a really strong work ethic. Right? And so and then they have an interest, an area of interest, and they apply that strong work ethic to the area of interest, and then you add time to that, consistency, and then they get something great. And I was like, this is one of the most basic formulas I could come up with. So I said, alright. I'm gonna start running today, and I'm just gonna, everything I do, I'm just gonna do with, like, excellence, clean my house, my room, and I just, and I was like, now what do I do? And I was like, alright. Now I gotta prove my life.

(Thomas at 00:48:24) No. You know, I went to the same awakening. It was I was a freshman or sophomore in high school. And I didn't know this because you come through an awareness. Right? As you read books and you study people, they all do the exact same thing. Somebody said to me it was my freshman year in high school. I was a smart kid. I did math. I did science. You know? I didn't tell anybody I did computer camp, didn't tell anybody in elementary school. I wanted to know. But, again, I just didn't, I didn't believe the thought. And someone said to me, you know, it just really changed my perspective. I would say changed my life. They said, your day is shot if you have one thing to do. And I'm like, what does that mean? What does that mean? I don't understand what that means. And I reflected on that for a couple days, and what it meant was that if I have something to do at, say, 4:00 in the afternoon, I'd sit around until that thing happened, and I'd go do it. And the awakening was I wasn't filling up my life with things. And to your point, make your bed, make the meal, make it the best meal of all time, mow the lawn. I mean, everything was, like, just go, consume, do, try, and my whole personality changed where I just got excited about doing anything and filling up my day with things to do. And it changed my whole viewpoint on life. So to your point, is you see these people that are just natural that way. I say I was a natural, but didn't know how to be.

(Joel Beasley at 00:49:48) Yes. The power of information, right? The awakening word is the best word I found to describe it. It's like I just turned on. I had more awareness, and then the goal became, how do I persist this feeling constantly? Because things that will throw you off are like you get sick, right, and you have to get back on the horse after you've been knocked out for a couple days. But when you improve the health and your food—I can cook really, really well now. And, you know, why couldn't you, why shouldn't you be able to—you don't need a lot of money to cook well. It's just you need the knowledge and the experience, and it's become a muscle. And it's like, now I—I mean, my daughter gets this. So we ordered gluten-free pizza a few weeks ago, right? And she didn't want to eat it because she had never had something out from order come into the house because we just make all our food.

(Joel Beasley at 00:50:48) And she's like, it's not the same. You know, she's three, so she was just like, no. But that made me feel good because for her to think that the food comes from the house, to me is a pretty exciting thing.

(Thomas at 00:51:02) Well, you know, it's funny. People say, why don't you get tired, right, with all that energy you're putting in? Why don't you get tired? And what I've learned through my experience is the worst experience is having that void of nothing to do. You get so used to it. It's almost an addiction of information and trying that the moment you take a pause, you feel lonely. You feel uninspired, and you need to jump into something else. So if something—you know, let's say you finish something—when you finish something, it's some of the saddest moments that I have. It's like, uh-oh, I have no purpose. I need another goal. And the joy of having that next goal, that next goal, that next goal, it really fills you. And people get addicted to it all around you. So when you walk into a room, people are like, oh, Tom has a lot of energy because I want to hear what they're about. So tell me about yourself. What are you interested in? And then when they say it, I'm like, well, that's cool. Let's talk about it. Well, let's discover any possibility within that.

(Thomas at 00:52:03) So it's not just about us. It's about them. And the ability to listen and hopefully inspire them to, well, you should do this because that's so interesting. And it's just a great way of seeing the world versus the classic saying, half full or half empty. You know, if you think it's impossible, you'll never do it.

(Joel Beasley at 00:52:21) That is—I fully agree. And you brought up a thought: When do you think we're going to get back to meeting in person, the business world?

(Thomas at 00:52:32) So, you know, I struggle. I'm a people person. I know it's our new reality, right? So I feel like I've gotten to know you really, really well within, I don't know, an hour.

(Joel Beasley at 00:52:43) Yeah.

(Thomas at 00:52:44) And I find with high-tech, because of all this remote communication, we're used to it. I hope sooner than later, to be frank. I think, you know, there's a thing I heard about social debt where we're paying it down, where the reason why things are working so well remotely is because we built up all this credit of social awareness. We know each other through our social awareness. I don't know if that's true or not, but, you know, as an office space, I know everybody. So there's not that extra time spent. I hope it's soon, but I love hearing about problems almost accidentally where you can help without having to reach out saying, how are you feeling today, right? Versus seeing somebody that maybe needs some help, but they didn't know how to bring it up or they weren't sure they needed help. That's where I think this remote communication is lacking.

(Thomas at 00:53:42) But on the other side, people said they've been more productive than ever before. So it's hard for me to say when this will—well, but I'm in Boston, so, you know, I haven't flown forever now. I feel like I don't live anymore. But when you go outside, you feel alive. So I don't know. Hopefully soon.

(Joel Beasley at 00:54:03) Yeah. I was thinking, you know, probably a couple months before—there will always be a certain—I'm curious, when will the 80% be doing it, right? Because there'll be some people I don't think will ever meet with people again.

(Thomas at 00:54:16) No. I talk to those people. There are people that will stay home for the rest of their life now. They were always hermits. They love it. They're good. I think human nature, you hear this, right? Social interaction is how we become human. And I think that's how we thrive. I think there are a majority of people that they just need to be healthy and mentally healthy, physically healthy. I know that I packed on a few pounds because I was sitting around too much. I need to start getting out, start moving around.

(Joel Beasley at 00:54:45) Yeah. I think it'll be a couple months until we're doing the business thing in person. But I don't mind. I mean, we built this whole podcast and this whole company through video calls. And yeah, it worked out. It's working out pretty well. I say that like it was a smooth path, but we are alive and, you know, we're growing. We had some interns this—right now. So we have three or four interns, which is actually pretty cool, very useful. Get different perspectives, get some fresh energy in a core team. Yeah.

(Thomas at 00:55:16) That's funny. And I'm assuming they are remote as well? And so I brought on my first time a truly remote person when the pandemic was just happening, and I wasn't sure. Now this turned out that I hired the right person, and they just clicked in. Because I spent so much time—I bet on people. I don't bet on necessary knowledge, right? And this person just clicked in, you know, wonderfully. And you can always teach somebody something, you know. But do they know how to work in a team? Do they know how to communicate? Do they know how to reach out? Those are all key things that I'm learning myself. There's a new learning of how to do this full-time, and I think we're doing pretty good at it. But, you know, having an outdoor drink on the water, I like that too with friends.

(Joel Beasley at 00:56:10) Yeah. And I think a couple of things. I think, first of all, this technology is obviously going to get better to where I believe in the future it'll be like we're in the same room. I think we're not far off from those improvements. And yeah, so that is exciting. But I also like the new mix up because I come from a mostly remote background, and the benefit of being able to structure my day where I can do my breakfast, go to the gym, get a work block done, go do this, get a work—I could keep moving versus being stuck in an office or whatever. There's a lot of benefit. It's not right for everyone either. But if you know how to do it and you're comfortable with it and have experience and you can manage your life and you're at that type of level, then it's really just all about bringing value and achieving results. And if you can manage yourself, then you're a very valuable player.

(Thomas at 00:57:07) Yeah. You know, it's funny. I think for those who, like us, who are just so driven, you know, you don't need to have someone tell you to work. You just naturally do it. My team is just that way. The one thing that I miss is whiteboarding. Yeah. That is—you know, sure, you can draw on someone's screen, but having two people draw, three people draw on a whiteboard, the speed of information exchange is so high, so intense that—not that you can't do it—there's a joy in that that I do miss.

(Joel Beasley at 00:57:39) Yes. So when I used to talk a lot about the remote or in person, I'm a big fan of when you're solving ridiculously hard problems, be in person with a whiteboard. Because for me, that's just been one of the easiest ways to do it. Keep in mind that 99% of my career I've been remote. But being in—I can't speak right now. When we were—I get excited—we were working on a project. I was flying out to California one week a month because we were building something that had never been done before, and it was difficult, and we all needed, you know—we're all remote. We all met in one central area, rented a hotel and a conference room in a hotel, and we would just burn through the most difficult parts of the idea over several months during that one week. And that was a very useful way to do something really difficult that was new, you know?

(Thomas at 00:58:32) You know, and also the joy. I just—you know, we talk about what can work. A lot of things can work. We can always make things work, but there's a joy in being in the same room with another human doing that work. And just like playing sports, right? You can play video games remotely, and it's enjoyable. But if you're on the field together, there's a whole different experience, and that's a joy and enjoyment too. Going to the game, you know, yourself and watching it there versus on TV. These are all things that have perspectives that one is not perfect on either side.

(Joel Beasley at 00:59:07) How do we get people to experience Chaos?

(Thomas at 00:59:12) So a lot of people are building their solutions or having companies in the cloud like Amazon. Amazon's first service was cloud object storage S3. So if you're storing data in cloud service like S3, come to our website, you click free trial, you're up and running within minutes, and now you're doing analytics on your data. You own it. It's a five-minute exercise. And the cool thing is you could go with us. Maybe you have one bit today to analyze, but you can scale to petabytes within, you know, a day, right? So that's a powerful thing with Chaos. So come to our website, chaossearch.io, and sign up for a free trial. We allow you to kick the tires, play with it, and really focus on your real side of the business, not the operations.

(Joel Beasley at 01:00:01) That's exciting. Chaossearch.io. We'll put links in the show notes and to all the awesome content. You're a great writer, by the way. Thank you. It's fantastic to—I've, you've got a new subscriber, a new fan of your content. So—

(Thomas at 01:00:13) So my next blog article is Multiples of Small. So I'll send a link to yourself, Joel.

(Joel Beasley at 01:00:20) Yeah. Please do. We'll actually—whether it's a couple weeks or whenever we do get it—we'll append it to the show notes so that people can reference it. Great. Awesome. Thank you, Thomas. I appreciate it. Thank you so much. Cheers. Great meeting you. Talk soon.

(Thomas at 01:00:35) Definitely. Bye-bye.