AI Technology in the Legal Industry with Rawia Ashraf, VP, Product, at ThoughtTrace, part of Thomson Reuters

(Joel Beasley at 00:00:03)
Hello, my friends. Today we're talking to Rawia, VP of Product Legal Technology at ThoughtTrace, part of Thomson Reuters. And we discuss how AI document understanding technology can save hundreds of hours and thousands of dollars in the legal industry, how to encourage a healthy work-life balance for your team, and how to be aware of people's different styles of working within your teams. All of this right here, right now on the Modern CTO podcast. Here we go.

(Intro Narrator at 00:00:33)
This is the Modern CTO podcast.

(Joel Beasley at 00:00:42)
I think over a year ago, we did an episode with the CTO of ThoughtTrace. So for those that might not have heard that episode, can you give me the overview of what ThoughtTrace does?

(Rawia at 00:00:53)
Sure. Yeah, ThoughtTrace is a document understanding AI platform, really well established in the oil and gas, sort of energy space. And what they do is help users analyze hundreds or thousands or millions of documents for key information. So they build models that can extract the type of data and information folks are looking for to understand out of their documents. At Thomson Reuters, we're very interested in those sorts of things, and so we're in the legal space. And really our goal is to help ThoughtTrace scale into the legal space and solve those same kinds of problems for our legal professionals, our users, that ThoughtTrace originally can solve for oil and gas users or business people.

(Joel Beasley at 00:01:36)
Yeah, I remember the time saved metrics were one of the craziest parts of hearing about that because, I mean, it takes a while for people to read through documents and pick out what they need. Can you refresh me on those metrics? I just remember they're crazy.

(Rawia at 00:01:54)
Yeah. So I'm not exactly sure which ones Joel shared with you, but something that we just learned recently, we had a lawyer using ThoughtTrace as part of a due diligence review. And he said the amount of time ThoughtTrace saved was in the hundreds of hours. And in terms of what that would cost them for outside counsel to do that work would be thousands upon thousands of dollars, right?

(Joel Beasley at 00:02:15)
Yeah, that's billable hours.

(Rawia at 00:02:17)
Yeah, exactly. And they were able to solve this problem. I think he said it took them twenty minutes to get it done, right, that would have taken hundreds of hours for outside counsel to do that work. So it's meaningful.

(Joel Beasley at 00:02:28)
Yeah. Yeah. So what's your role at ThoughtTrace? And actually, can you give me a little bit of your background and career journey?

(Rawia at 00:02:35)
Yeah, I'm a lawyer by background. I practiced antitrust law or competition law for a little less than a decade in both Washington, D.C. and New York at a global large law firm. And then I made the move to Practical Law, which is a service or a product that's really aimed at helping lawyers know what to do in their job. So law school trains you on sort of legal concepts, but it doesn't really teach you how to do your job, you know, as a trade. So how do you draft a contract? What are the things you should be thinking about in this? You're going in front of a judge—actually, what do you say? How do you prepare yourself? So it's the know-how piece of law. So I joined Practical Law to help launch their practice area on antitrust because I practiced antitrust law. And that's really Practical Law's model, is to hire experienced subject matter experts in a particular area and have them build out the content for sort of their peers. I did that for a number of years, and then I moved over to the product management side of Practical Law, or our business strategy and our product development and our investment strategy. Did that for a while, and that's, I think, where my heart is, is in product management. You know, I really found my space. And I've been now for the past few years working in product management on the legal technology side, so around contract analysis and document automation and those sorts of parts of lawyers' day-to-day workflow. And I just recently became the Vice President of Product for ThoughtTrace.

(Joel Beasley at 00:04:02)
Nice. Yeah. So do you have a background or history of being interested in tech for fun? Or if not, what's the learning curve been like becoming VP at a tech company?

(Rawia at 00:04:13)
Yeah, I think I've always had a background or interest in problem solving, and I think that's the connection for me. I definitely, you know, didn't grow up coding or building robots or those sorts of things that a lot of my colleagues have done. It's been much more on problem solving and understanding jobs to be done and sort of, you know, what are people's priorities? And so that's kind of what brought me into product and closer and closer to tech. In terms of the learning curve, yeah, for sure, there's been a learning curve. But I've been lucky to work with a lot of smart colleagues, you know, and just sort of be exposed to what they're doing and taking my own initiative to study and learn. But yeah, I guess I've sort of just backed my way into it.

(Joel Beasley at 00:04:52)
That's awesome. So what's your day-to-day look like at ThoughtTrace?

(Rawia at 00:04:57)
Yeah. So, as we just mentioned, Thomson Reuters, which is where Practical Law is and the products that I've worked on, acquired ThoughtTrace. And so a lot of what we're focused on right now is launching ThoughtTrace into the legal professionals market. So as I mentioned, it's a great product in the energy space, and we want to bring it into sort of the legal landscape overall. And so we're doing a number of things. We're getting ready to launch to law firms, so I'm kind of working on that piece, talking a lot to our law firm customers around their needs, their tech stack, and how ThoughtTrace fits into that. We've already launched into corporate legal departments, so that's a big part of what we're doing. And then I think maybe the most exciting part of what we're focusing on is, how do we build the best models, the best AI models in the world for legal professionals? That's at the heart of our project. And as I mentioned before, I worked at Practical Law, and Practical Law is full of subject matter experts who are deep experts in their areas of law. And so really what we aim to do and what we're doing is bringing Practical Law and ThoughtTrace together. So getting our Practical Law subject matter experts to leverage the AI building capabilities in ThoughtTrace to build out very specific, very deep domain models in different areas of law for our users. So finance, capital markets, M&A due diligence, insurance, IP, all of those sorts of topics. We're taking experts in that area and using the powers of ThoughtTrace to create new models as content types, almost, for our users.

(Joel Beasley at 00:06:28)
That's awesome. So I read—and I think you started to touch on it there—but I read that you were working on creating the standard for legal AI. What does that mean?

(Rawia at 00:06:38)
Yeah. So contract analysis has started to make inroads in the legal technology space, right? It's one of the areas. There's innovation and experimentation and maybe some customer adoption, but it's not widespread. And it's really our belief that users don't want to spend a lot of time training their own AI models, right? And for sure, there are people that are lawyers who are interested in technology and interested in AI, and that's fine. But as their day-to-day job, they don't want to be building AI models. And what we aim to do is take that pain point away and really start thinking about legal AI as a service. Let us build the models. We have the expert lawyers. We have the technology to do it. So we'll build the models that will set the standard for, you know, analysis in the legal space.

(Joel Beasley at 00:07:24)
So I imagine when AI started making its way into medicine, there had to be a period of building up trust. Like, is this computer going to look at my brain as good as a doctor? Because this diagnosis needs to be correct. There's also similar high stakes looking at a contract. You don't want to miss anything.

(Rawia at 00:07:43)
Absolutely.

(Joel Beasley at 00:07:44)
Are you guys doing education within the legal space to try and start building up that trust for prospective clients?

(Rawia at 00:07:53)
Yeah, definitely. And, you know, Thomson Reuters has long been a player in the AI space. But what I will say is that I've seen a curve sort of developing where lawyers are starting to gain trust in AI, right? It's very different from where it was, I would say, three or four years ago. There's an understanding, you know, and I think also an appreciation of what the AI can do and what it can't do, to understand that there are limits to it. I think if people understand those expectations, then they can sort of develop that trust in the AI. But it's not part of our vision to remove lawyers from the loop, right, and have, you know, AI-driven contract analysis that eliminates the need for lawyers. It's really to help lawyers focus on the things that are most important in a deal or in a matter and using the AI to triage that and sort of take a really big problem and make it a lot smaller so that the lawyer can focus on that and really add their value and their judgment to that smaller problem, as opposed to looking through 20,000 documents to find a couple important pieces of information. Let the AI do that piece, right? And then you apply your judgment on what that information means and how you're going to advise your client based on that.

(Joel Beasley at 00:09:01)
Right. Yeah. It's not doing the analysis and coming to conclusions, right? So from what I'm understanding, the capabilities is quantifying, I guess, and breaking down the valuable pieces of the contract so that they're easy to look at and aggregate. And then that's where it ends?

(Rawia at 00:09:23)
I would say that's where it starts.

(Joel Beasley at 00:09:24)
Oh, nice. Okay.

(Rawia at 00:09:26)
So ThoughtTrace will narrow that problem down for you, do the reading on your behalf, get you to sort of the important information. And then from there, we enable you to sort of apply judgment and say, "Okay, this is the extraction. This is the clause that ThoughtTrace found in this document. Here's, as a lawyer, here's what I think," and apply your analysis to that. And then continue, and then share that with your colleagues or move that into a report or add that into a workflow for how you do your work. So it's one piece of where ThoughtTrace starts on the journey.

(Joel Beasley at 00:09:54)
That's cool. So tell me about what is "one plus one equals 100"? That's something I've seen in the branding.

(Rawia at 00:10:02)
Yeah, it's the way that I see Practical Law subject matter expertise and ThoughtTrace coming together, right? So the reason Thomson Reuters acquired ThoughtTrace is because it has the best model building capabilities in the market, right? We did a huge market scan. We looked at a lot of other players. We dabbled in the space as well with building tools. And what ThoughtTrace can accomplish in a short amount of time is just unparalleled, right? So if you think about that, sort of the model building capabilities in ThoughtTrace from an AI perspective, and if you marry that up with, as I mentioned, the best, world-class subject matter experts—really, those two things are our differentiator. You know, there's a lot of players in the market. No one has that, sort of the best market-practicing lawyers to guide and develop the AI. So that's really the one plus one equals 100. It's really just taking legal AI to a whole different level and taking that pain away from lawyers and law firms of having to train their own AI. It's really that equation.

(Joel Beasley at 00:11:05)
Yeah. So your subject matter experts that you've had for years are the ones training the AI.

(Rawia at 00:11:10)
That's right.

(Joel Beasley at 00:11:11)
Okay. That's cool. That kind of reminds me, another medical reference just because we've had a couple of those companies on the show. There's this one called Intelligent Medical Objects that does AI for understanding medical terminology, and they also codify a lot of medical terms. And they're talking about deploying terms to hospitals, which is something I didn't realize needed to happen, but it makes sense so that they're standardized.

(Rawia at 00:11:39)
Yes. Yeah. And it was kind of a similar idea around, they have doctors and specialists that, instead of their job being doctor, their job is at Intelligent Medical Objects codifying terms because they're the best people in the world to be doing that.

(Joel Beasley at 00:11:56)
Yeah, that's exactly right. And it's the kind of thing that when you're in practice, I imagine, as a physician or in practice as a lawyer, you don't have time to sit down and do those sort of academic things of create a list of terminologies or, you know, build out the perfect way to approach a contract. But that's the kind of stuff that folks at Practical Law, Thomson Reuters, can do. That's their full-time job.

(Joel Beasley at 00:12:19)
Yeah. And so when I'm talking about AI with someone, I always like going to the ethical side of it and how are you avoiding bias in that conversation. And I'm especially interested here because, I mean, legal stuff, obviously, huge impacts to people's lives. Can you tell me a little bit about the steps taken at ThoughtTrace and Thomson Reuters to make sure bias is staying out?

(Rawia at 00:12:45)
Yeah. Well, it's interesting. So some of the big issues that, when I think about bias in legal AI, it's much more sort of when things are predictive and based on prior case law or sort of sentencing, right? So for example, you know, you've seen things where if you use AI to predict the likelihood of recidivism, right, of a prisoner or a criminal, the AI will, because of what it's been trained on, will recommend a higher sentencing for a Black person as opposed to a white person. And it's very divorced from the actual performance of those two individuals, right, when it's based on historically biased case law or judgments going into it. So that's not an issue really that we have to deal with in the types of models that we build, right? Our models are more trained to say, if you have a contract and the contract has an indemnity clause, right, help me find similar indemnity clauses. So there's, I mean, there's risk for bias in the way that one person might interpret indemnity in a different way than someone else would interpret indemnity, but it's not the sort of prejudicial bias that impacts someone's rights, right? So not to say that it wouldn't be there, but it's actually much more legal interpretation as opposed to sort of individual rights. So that way I feel pretty good about what we're doing. But I totally hear the point, you know, and ThoughtTrace, obviously, at TR, we take lots of steps to make sure that we have a variety of training data in what we do. But I think also equally important is that you have a diverse set of people doing the training and a diverse set of people evaluating the training, you know, in any type of supervised or unsupervised learning. I think it's sort of not only diversity in the dataset, but diversity in the people who are involved in the project.

(Joel Beasley at 00:14:27)
Yeah. Absolutely. That's huge. I just remember when conversational AIs and stuff were first coming out, and they're just trained by a bunch of dudes in a lab. And they definitely had their shortcomings and still do. I mean, there's a lot of work to be done.

(Rawia at 00:14:42)
Oh, for sure.

(Joel Beasley at 00:14:43)
But it makes a lot of sense that as ThoughtTrace is acting more as a natural language understanding for legalese, it's—

(Rawia at 00:14:53)
Yeah. Yeah. And I think it's really important for the end users of AI to understand how the models that they rely on or the algorithms that they're relying on were created, right? So that's something that we'll be transparent about. We like to really talk about that with our customers as well. Sort of, what goes into building these models? What types of documents is this trained on, right? How is this tested? How is it evaluated?

(Joel Beasley at 00:15:16)
Yeah. So where is the human in the loop on the training process? The subject matter experts are training the AI, but I'm not exactly sure what that means in context. Are they selecting the data or looking at the inputs and outputs or both? What does that look like?

(Raviya at 00:15:34) All of that, right? So let's say we want to build a model that helps you analyze a software sales agreement, a SaaS agreement. The way that we would do that, or the way that we do do that, is we have a Practical Law subject matter expert who spent a lot of their lives drafting SaaS agreements and negotiating SaaS agreements.

(Raviya at 00:15:52) They sit down and they start to say, these are the types of things that are important in the SaaS agreement. If I were reviewing a SaaS agreement, if I asked a junior associate in my firm to review a SaaS agreement, these are the things I'd want them to look at. So they start out by building that. And you can call that an ontology or taxonomy, but these are the types of relevant concepts in the agreement.

(Raviya at 00:16:12) Then what they do is then we take a bunch of SaaS agreements and annotate them towards those concepts. So if they say a super important thing in a SaaS agreement is the length of the contract, then we'll find 50, 100, however many diverse examples of that to train the AI to recognize length of term in a SaaS agreement.

(Raviya at 00:16:32) And then we test it against agreements, agreements that the machine hasn't seen, that it hasn't been trained on. Either more documents in the wild or customer documents when they test out models to see how it performs, and then we fine tune and calibrate. So the human is doing all the training.

(Raviya at 00:16:48) What we've taken out of the equation is really having to have a data scientist then take the training and tune that to build the model. That's already been automated in ThoughtTrace's ML platform.

(Joel Beasley at 00:17:00) So is that difficult data to procure? I imagine companies are probably a little bit protective of their contracts.

(Raviya at 00:17:08) Yeah. So there's a huge publicly available database of contracts in the United States, agreements that are filed with the SEC. So that's publicly available. And Thomson Reuters, because we are legal content providers, we have a lot of access to legal information. So that's another strength we bring to the table.

(Raviya at 00:17:25) And then ThoughtTrace historically has had, for their corporate customers, the access to the corporate documents to train models. So it's really—it's seen a very diverse set of content in training.

(Joel Beasley at 00:17:37) That's awesome. That's smart. So something that's been on my mind because I've been feeling it and working on with my team is burnout. I somewhat recently got to do an interview with Phil, a CTO of a consultancy called Melillo, and he talked about never wanting to overload their employees. Because as a consultancy, sometimes there's big fires that need to be put out that require a lot of overtime, so everybody needs to be kind of not stretched thin the rest of the time so they have margin.

(Joel Beasley at 00:18:08) And yeah, it's strategic to give margin for his consultants so that they're able to put the customer first when they really need to. And I'm just curious, on your team, how do you gauge how they're feeling with their work, and how do you try to prevent burnout for them?

(Raviya at 00:18:28) Yeah. It's a great question. I know lots and lots of leaders and teams are trying to figure this out. I think this might sound obvious, but to me, the most instrumental thing is to talk to your team and find out how they're feeling and get honest answers out of them. And you can only do that if you've invested the time in building the relationship with your team. So it's not something that you can start now. If you've invested the time in building relationships with your team and you have that trust, you can talk about how they're feeling, how they're doing. So I mean, I make that a key part of how I relate to my team.

(Raviya at 00:19:00) I think the other really important thing one can do as a leader, as a manager, is helping their team prioritize. Because everybody has more than they can do. Very few people have margin protected for things. And so if you can help by setting clear priorities for your team, but then also when they're dealing with a workload, helping them strategically prioritize within that and deciding what's not going to get done, I think that is critical. We all know here are the things you need to do, but we have to be able to say here are the things I'm not going to do because I want to have time to do these things and do these things well and not burn out. So to me, those are things that my managers have done before that have been helpful, and I think those are the things I try to do with my team as well.

(Joel Beasley at 00:19:43) That's awesome. Is that stuff you handle in one-on-ones, or do you have open discussion ever?

(Raviya at 00:19:50) Yeah. I think it's a combination of both. So I think in terms of setting priorities for the team, that should be pretty widely and publicly done and shared. So it's either in an all-hands meeting or your team meetings, creating documentation around that, OKRs. So everyone knows what are we trying to do? Where are we going? And there should be a collective understanding. So if anyone has to make a decision about what's important, where to focus their energy, they can refer back to those priorities. Does this further those priorities, or is this work taking me in another direction? And I don't need to go down that path.

(Raviya at 00:20:19) But then in terms of individual workloads and individual priority setting, I think that mostly happens in a one-on-one setting. We talk through, what are the things you're doing over the next two weeks, in the sprint, for example?

(Joel Beasley at 00:20:33) Yeah. That's something I've been learning recently, helping with the individual workloads of my team and helping with prioritization. Because some people just will ruthlessly prioritize. You just say, hey, here's the things that you need to get done. Tell me when they're going to happen. And then you get dates, and that's when it happens. But some people are—when you say, here's all the things that need to get done, they're like, oh my God, I have to get all these done right now.

(Joel Beasley at 00:21:01) I'm going to work until 8 p.m. And I've been learning to recognize that so I can say, hey, don't work until 8 p.m. I just need to know when. I don't need it to happen today.

(Raviya at 00:21:12) Yeah. You know, another thing on that point, what we talk about a lot is the shadow of the leader. So if you're sending your team an email on Sunday, and even if you say, hey, sorry for the Sunday email, don't worry about this till Monday, they're going to worry about it on Sunday.

(Raviya at 00:21:28) And I'm totally guilty of that because I always did that because I'm thinking about work on Sunday, sitting waiting to pick up my daughter from something, and I'll shoot an email to the team. And then I realized that as much as I say don't worry about this, I've now put work in their face on Sunday. So stop doing those things. Stop kind of—set examples for the kind of behavior that can lead to work-life balance, I think.

(Raviya at 00:21:51) And that's, of course, challenging in a product organization or with a motivated, ambitious team, but I think that's an important thing too. And that's something I need to keep working on.

(Joel Beasley at 00:21:59) Yeah. I had a realization of that too because I would do the after-hours messages and stuff. But whenever I would get an email after hours when I'm doing something else, it would just ruin my day.

(Raviya at 00:22:11) You're frozen in a loop, right?

(Joel Beasley at 00:22:13) Yeah. Yeah. So definitely schedule send was a—

(Raviya at 00:22:18) Game changer.

(Joel Beasley at 00:22:18) Yeah. Game changer.

(Raviya at 00:22:19) But then the other problem is that Monday at 9 a.m., you get all these emails that everyone's scheduled over the weekend to go out. They all kind of flood in at once.

(Joel Beasley at 00:22:27) It's not perfect, but I think it's better.

(Raviya at 00:22:30) I agree. I agree.

(Joel Beasley at 00:22:32) So what would you say is something you wish you knew when you first transitioned from practicing lawyer to leading a team?

(Raviya at 00:22:43) A couple of things. I think maybe one is to be aware of everyone's tendency, I think, to gravitate towards people who are like themselves, whether that's your demographic profile or it's your ways of working or your sense of humor and stuff like that. You know what I mean?

(Raviya at 00:23:02) And I think I have noticed over time before that I've done things like that where there's certain people I will go to more and work with more just because there's something about them that's familiar, and it's probably familiar to me. And so to be really conscious of that from the outset and make sure you are equally distributing work and opportunities and your time to your team, I think, is something that I'm very conscious of now, but it wasn't even on my radar going into being a people leader.

(Joel Beasley at 00:23:29) Yeah. Definitely. It's hard not to give extra attention to someone if you like talking to them.

(Raviya at 00:23:36) Exactly. Yeah. Exactly.

(Joel Beasley at 00:23:38) Yeah. And then that's really important because that affects everybody's money and life.

(Raviya at 00:23:44) Yeah. I mean, I think the other thing is actively making time to coach people on your team and understand where they want to go with their careers and give them opportunities to get there. So to have those proactive conversations. I've not been super directive with my career. I kind of want to work on interesting problems and work with cool people, and so I sort of go where things take me. And I think that as a manager, you can do better by your team to—if they do have ambitions and they do want to go certain places or they just want to develop certain skills, how do you give them those opportunities, and how do you coach them and push them into those things?

(Joel Beasley at 00:24:21) Yeah. That's something that my cohost Joel—and by the way, when your company's Joel was on the podcast, he was interviewed by my company's Joel, which is funny.

(Raviya at 00:24:31) Yeah. Joel to Joel.

(Joel Beasley at 00:24:32) That's something that my Joel did pretty well for me as I've worked lots of different roles within our company. Every time I was needed to do some other change, he asked, is this something you want to do? Do you have any interest in actually leading a team or something? And I know that if my answer is no, that's going to be okay, and we'll find another solution. That's been a really big benefit for my work environment, mental health, just knowing that I do have autonomy here.

(Raviya at 00:25:06) Absolutely. Absolutely.

(Joel Beasley at 00:25:08) Before we wrap up, is there any extra shout-out or soapbox you want to step up on before we wrap up here?

(Raviya at 00:25:16) Well, I mean, I definitely want to send a shout-out to the ThoughtTrace team who joined Thomson Reuters over the past couple months, and it's been wonderful for TR. We're super excited about this team and the product and the opportunities ahead of us. So I'd be remiss to not shout that out before we go.

(Intro Narrator at 00:25:34) Thank you so much for listening. And if you found this episode useful, please share it with a friend or colleague who you think would get value from it. And if you have topics that you'd like to hear discussed on the podcast, either add me on LinkedIn or send me an email, [email protected]. Every time I get an email or LinkedIn message, it absolutely makes my day and inspires me to keep going.