You don’t want to miss the 27th episode of the Alternative Litigation Strategies podcast where we discuss one of the most innovative and transformative legal technology platforms to hit the litigation marketplace in the last several years – Optimal Legal Audio (OLA). In this episode Certum Group Director, Kevin Skrzysowski, interviews Adam Feldman, a lawyer, political scientist, professor, Supreme Court scholar, and creator of the Empirical SCOTUS Blog, and Badri Narasimhan, an engineer, serial technology entrepreneur, and all-around analytics wizard. Adam and Badri discuss their latest invention – Optimal Legal Audio (OLA) – an artificial intelligent (AI) platform that performs audio and textual analysis of oral arguments to predict judicial decisions. OLA analyzes elements such as word choice, word volume, tonal analysis, and interactions with the bench, to successfully predict votes in State courts, single judge trial courts, the U.S. Supreme Court and more. OLA helps counsel by steering them in the right direction on specific interactions with judges or Justices allowing them to better inform their clients of potential case outcomes. OLA is truly a game changer in Appellate Law.
This transcript has been lightly edited for grammar and clarity.
Kevin Skrzysowski:
Welcome to the 27th episode of Certum Group’s podcast, Alternative Litigation Strategies where I interview esteemed members of the bar from top law firms, companies, and consulting firms across the country. On this program, we discuss the latest litigation trends, strategies, and technological developments across the litigation marketplace. I’m your host Kevin Skrzysowski, a director with the litigation consulting firm Certum Group where we specialize in working with businesses in our outside counsel to mitigate cap and transfer litigation outcome risk. I would like to note that just last week, the social media aggregator and ranking organization, Feedspot named this program one of the top 10 must-follow litigation podcasts of 2024.
That’s because we have really exciting content and super smart guests, and that trend certainly continues today. Today, we have a really exciting program because we’re going to discuss what I think is one of the most innovative and transformative legal technology platforms to hit the legal marketplace in the past year or perhaps the last couple of years actually. And it’s called Optimal Legal Audio or OLA, and it was created by my two guests today. I am pleased to be joined by lawyer, political scientist, professor, Supreme Court scholar and creator of the Empirical SCOTUS blog, Adam Feldman, and engineer, serial technology entrepreneur, and all-around analytics wizard, Badri Narasimhan. Adam and Badri, welcome to the program.
Adam Feldman:
Thank you for having us.
Kevin Skrzysowski:
Absolutely. So gentlemen, I noticed that just this week on the Empirical SCOTUS blog, you posted an article that you’ve been collaborating over the last several months to launch OLA. I think it would probably be smart just by starting to have you tell us what is Optimal Legal Audio.
Adam Feldman:
So this is an idea that was kind of percolating in my head for a while, how we can look at oral arguments and use that as a point of departure where we can actually get some strong predictive cues on how cases are going to resolve, specifically looking at how judges and attorneys talk to each other and looking at a case as a unit of analysis where we can get signals from judges about how they’re likely to vote. So previous technologies in the judicial behavior space that look at predicting outcomes have been primarily based on past data and trend analysis. And there have been a few studies looking at oral arguments, but most of them have been backward-facing. And so we’ve taken a lot of the tools that have been in practice, combined them, developed some innovative ways of thinking about this that haven’t been out in other platforms and we’re one of the first, if not the first group that has started looking at this from a predictive standpoint. So looking at things that have yet to be decided and are testing it out on a daily basis on new sets of cases.
Kevin Skrzysowski:
I’ve certainly never heard of forward-thinking predictive analytics for oral arguments before. I think this is a first for the market. You mentioned several different metrics or items that you’re measuring during the course of the oral argument. Tell us, what are some of those? How does OLA work?
Adam Feldman:
OLA works on both text and the actual audio, and we’re integrating them both under our algorithm, which takes the words that judges say and how they say them across multiple attorneys and looks for a preference from one to another. At a really general level, what we’re looking for are emotional cues when the justices or judges, depending if it’s on the Supreme Court or lower court level, seem like they are becoming more focused on a party. So where their energy is spent asking more questions and the way that they’re asking them is actually using a tonal of voice that is getting more emotion. So with these multiple measures, we’re able to see some kind of preference that isn’t indicated in any part of the case prior to that before you actually have judges speaking directly to parties. So it’s kind of a one-shot deal and gives a very clear analysis of how the judges are likely to vote on the merits of a case.
Kevin Skrzysowski:
So the algorithm is literally picking up on word choice, word volume, word analysis, interaction from the justices to the lawyers and tonal analysis.
Adam Feldman:
Badri, is there anything else that we missed there?
Badri Narasimhan:
Nope, that would be a very good summary.
Kevin Skrzysowski:
Would you say that this technology that you’ve developed would fall under the category of machine learning and artificial intelligence?
Badri Narasimhan:
Well, there are elements of it that do and elements that do not. The typical definition of machine learning is that it learns from its own decisions over time, and this is not a learning engine in terms of the predictions. We do need human involvement to fine-tune the predictions, however, the transcript that is generated and the analysis of what is said is indeed using AI. But how it is said is an analysis based on an algorithm.
Kevin Skrzysowski:
Okay, very interesting. Now thank you for that clarification. So I’m assuming you’ve tested the platform for efficacy. Can you tell us a little about your testing and the results that you’ve seen today?
Adam Feldman:
I think you have those numbers probably more in front of you, Badri, than I do.
Badri Narasimhan:
Well, with Adam’s help, and I want to first make it very clear, I don’t know what… I once asked Adam, “What’s the P and the D next to a case? And he said, “P is plaintiff and D is defendant.” So that’s the level from which I started. But with Adam’s help, we’ve looked at supreme court cases, appellate cases, trial court cases, any place where there is audio, sometimes we only found video and we were able to play the video, convert the audio from there and use the system to analyze it. And the way we measured our accuracy is if there is a case where there are three justices, we would count how many of the three we got right. If there are nine, then we’d count how many of the nine. So overall we are batting it around 87, 88%. So approximately 300 votes and we have gotten give or take 270 of them correct.
Kevin Skrzysowski:
That’s a pretty good-
Adam Feldman:
I think it’s worth noting just talking about where those percentage numbers fall relative to the population of cases that have been examined outside of our work that in past we’ve seen that without oral arguments, that really strong predictions can predict about 70% of Supreme Court outcomes. With the audio, it’s picked up a little bit more from that in past trials maybe to the mid-70 percent, but we’re testing well at least 10% higher than anything else that I’ve seen on the market so far. So I think by combining these multiple measures and then thinking and fine-tuning how they’re weighted and relative to one another, we’ve been able to beat what the threshold has been so far in past tests.
Kevin Skrzysowski:
I think that’s an incredible statistic. Now is that your goal or are you planning to own the system to even increase that percentage more?
Adam Feldman:
So I think that’s dependent on interest. Right now we’re piloting the oral argument, that prediction in a vacuum and we’re doing quite well. Anybody that’s making decisions has a large litigation portfolio and where that marginal difference is important and they have some room to maneuver after an oral argument, we can talk about where we think this is going to have the most impact towards the end of this program. But in terms of what we’re thinking about doing is if we see interest and folks want to see this pushed up to even a higher level of accuracy, we could combine this with past data. And so right now we’re working with the tools that we have and from a computing standpoint, it’s easier to have it within a vacuum. We need great computing power to use past data to also run through the analysis. So it really depends, but we definitely can push beyond the line of accuracy that we’re at so far. It’s attainable.
Kevin Skrzysowski:
Very interesting. You’ve made a lot of comments about your technology and the distinctive assets and the value differentiation between your algorithm and what’s already existed in the marketplace. So let’s have a little bit of a discussion about the market and where you fit. And let’s start with, what led you to create this program? Or in other words, what are some of the historical approaches to understanding a judge’s lean?
Adam Feldman:
As a political scientist, I’ve explored the gamut of this and so prediction is kind of the gold standard within political science, I think within many of the sciences. Prediction is what we look for at the highest level. So if you push back far in the past, you had researchers thinking about the party of appointing presidents or federal court judges from district courts, appeals courts, and Supreme Court as voting different. So it really started by folks noting first, when there were dissents and then are the dissents in cases coming from judges that came from different parties of appointing presidents? And that was noted early on, we’re talking like ’50s and the ’60s. And so as quantitative metrics have improved, there have been movements that have looked at past data. And so data analysis looking backwards at what judges have done historically. And then more recently there have been studies on oral arguments and how those factor in through understanding how judges are going to vote.
So this started by researchers just counting the number of words and questions that judges were asking to the two parties. And so some of the first papers just counted those numbers and saw that there’s a strong correlation between the relative number of words that a judge says to one party versus the other party in a case and seeing that that led to a higher likelihood that a judge is going to vote against the party they speak more to. The why there has to probably do with a judge just not being sold on the position of the party they’re going to vote against.
So whether it is to show that that party has a weak argument or to maybe answer some of the questions they have about the things they find problematic, that’s kind of where this impetus of looking at oral arguments as something unique and different started. And since then there has kind of blossomed a series of different papers and different perspectives on how you can look at oral arguments as something unique in a case and some kind of tell that justices or judges are giving about how they feel about an issue that’s being resolved in the case.
Kevin Skrzysowski:
I definitely, I couldn’t agree more that data of drives decision-making. Lawyers rely on precedent every day to make decisions. But when it comes to technology and technology leveraging big data to help professionals including attorneys make decisions, I’ve noticed, because I’ve spent many years working in the legal information technology market, that lawyers say that they will embrace technology, but they’re normally pretty reticent to use advanced technology like AI and machine learning. Why do you think that is? And do you think there will be a diametrical shift given the advances that we have as Badri explained earlier in both machine learning and AI. Is this going to be an impetus? Are we going to see a change, a movement in the legal industry?
Adam Feldman:
I think I can talk about it from the legal angle and maybe Badri can speak to it more from the analytics angle. From law, I mean, as somebody who was a regular practitioner years ago, my foray into this was you start as a lawyer looking at doctrine. You do a lot of case research and you look at the precedent in a case you try to come up with the strongest argument that is supported by past cases. And law is a practice that is deeply entrenched in past ways. So you have kind of an old school mentality, especially at the top of law firms. So it’s top down type of corporate atmosphere. Now at the top you generally have these older partners who have the ways of doing business I think are reticent to try new things that are way outside of the box.
The other thing about big data that doesn’t always translate well to a lawyer’s mentality is lawyers are generally very focused on one given case and that cases that were in the same stream as the ones they’re looking at. So you’re not going to necessarily look out of jurisdiction, that makes total sense. But you also might be looking at 20 cases rather than 100 or 1,000. And part of the reason for that is lawyers generally don’t look at data sets. They’re looking at individual cases and the words of those cases. So what we can do is leverage thousands and thousands of cases rather than 10 or 20, that as a lawyer you can only read so many cases in a day. Well, using technology like natural language processing, machine learning and AI, we’re able to push those boundaries of looking at larger case sets and coming up with deeper analyses based on that.
But until lawyers become comfortable with technology, it’s going to seem like some insurmountable task to actually make sense of that. And there’s been headlines to that effect. One of the top articles that I’ve seen about AI in the law is actually about a legal case where individual attorneys used AI to try to create a brief and found that ChatGPT was giving false cases, because ChatGPT’s relying on crowdsourcing. So it doesn’t know necessarily a better versus worse source, it just knows what’s out there in the universe. So until you have a way of understanding and leveraging the power to best practices, it’s probably going to be a hard sell to have attorneys try to make use of it without having a background in the technology itself.
Badri Narasimhan:
Well, I would add to what Adam was saying from a different perspective, and my background comes from healthcare technology. The company I founded went on from my third bedroom to 27,000 plus customers. One of the main challenges, and we have the similar thing of the more
experienced position saying, “I don’t need technology. I’ve been doing this thing in a way that I’m used to.” And ultimately to a large extent, they were justified in resisting technology because after all this hype of billions and billions of dollars, physicians were handed technology that made them slower, not faster. Hold on a second. One more click. One more alert, one more click. And they just go, “What did I get out of this?” And very often things are a solution looking for a problem. In the healthcare side it was, convert your paper charts into technology and I hope you do something with it.
And right now we are all enamored by AI and NLP [natural language processing] and I hope it helps you. What it requires is really a partnership of people like the Adams of the world and you of the world saying, “I know law, but I need to think through ‘what are frustrations in my daily routine that I wish goes away?’” And whether we use AI or not is irrelevant. What you really are looking for is a way for your frustrations to be replaced by ease. And instead of taking AI like a hammer and hitting every problem with it hoping it is a nail, the real question to ask is, we are in 2024, litigation is increasing, not decreasing. The volume of precedent that you have to take into account is increasing exponentially.
How can I make decisions faster and better and really not care about how the cookie is made so to speak? All your IT teams accountable for, bring me a solution that makes my life easier. I’m not going to click more, I’m not going to type more. Teach me how I can make my life easier and partner with the right kind of people who can get that. And whether AI has a role or not is secondary. Maybe it does, maybe it doesn’t.
Kevin Skrzysowski:
Well, I think now I have a better understanding as to why you two gentlemen connected and created OLA, because the two biggest largest data sets that people are trying to wrap their minds around, not just people, but whole institutions and marketplace, medical information and legal information. So one more question for you and then I say we take a look at the program and then have a discussion about what your best market audience is and some use cases. So you’ve mentioned a lot about existing technology. What are some of the most advanced technologies that are out there today and how does OLA add to what there already is and what impact do you think that’s going to have on the market?
Adam Feldman:
I’ll tell you from the angle of a lawyer with a political science background that most of the technology that exists today has to do with legal research. So a big business move recently was Thomson Reuters taking up a Casetext. And Casetext has a feature called CoCounsel, which helps with brief-writing and it has the possibility to really pick up the pace by filling in some of the gaps that attorneys might have that they would otherwise spend from minutes to hours doing legal research. It’s advanced technology leveraging AI to speed up the process. And so from what I’ve seen with technology today, that’s even one of the first movements towards using an AI-based technology in helping with the research process in litigation. I would actually say that the first set of advanced technologies that came into law had to do with the discovery process and how to speed that up.
Can we use machine learning to get through a million documents faster than it’s just being hand coded? Which when I was in a law big law firm practicing, we would spend hours and hours just combing through documents looking for a few keywords within them. So technologies enhance that pace. Not a lot of people have been looking at the behavior of judges and trying to understand how you can use back-facing data or current data to predict what’s going to happen down the line. So we’re really, I would at least argue that we’re on the cutting edge of that movement of trying, not looking at necessarily the language being used to make an argument, but rather at anticipating how a judge is going to rule based on whatever priors that we have to understand that.
So I think we’re coming at it from a very unique angle where we’re looking at prediction and not at adjusting an argument. So I don’t see a lot of competition in this area, but I also see there not being a market created because there really is no existing technology in this space. So we’re really looking at individuals that see themselves as innovators that want to adopt something new that could potentially save a lot of thought and heartache that goes after an oral argument and waiting for a decision.
Kevin Skrzysowski:
That’s excellent, Adam. Like I said at the beginning of the program, I think that this is innovative and is truly transformative. As far as I know there’s a lot of blue ocean out there for you gentlemen, because nobody is looking at predictive analytics and the behavior of judges. So can we take a look at the system and look at some of the behavior of the judges and some of the interactions between the judges and the attorneys themselves?
Badri Narasimhan:
I’m going to share my screen and we’ll kind of begin with a little bit of hot news in the media this morning. This is the mifepristone case, and this was the verdict was announced today and there was a 9 and 0 unanimous verdict where the FDA, the petitioner got their way. And so mifepristone is staying in the market and this is OLA and kindly note the date here it is May 17, 2024, almost a month ago, and we predicted 8 and 1. And I’ll stop right here and have Adam explain what happens sometimes in the back room, because these are things I learned from him. Adam, if you want to explain more.
Adam Feldman:
Sure. So in the Supreme Court in particular, because you have nine judges on a panel and there are nine justices and they’re not necessarily making a final decision right after the oral arguments, sometimes you have movement. So it’s pretty much impossible without knowing what’s going on behind the scenes to predict based on oral arguments with absolute certainty, because of this movement after the fact. So past studies, and there hasn’t been this information out for a while, but in the ’90s there was a release of documents from Justice Blackmun that showed some of the conferences, they had notes from conferences about what happened after oral arguments. And so what researchers have found is that they’re just under 10% of the time that a justice will switch a vote after oral arguments and after they made their initial vote in a conference. And that generally has to do with one of two things, either that the draft opinions are drafted in a way that the judge or justice that thought they were on one side doesn’t agree with and then flips to another side.
The other possibility is that in talking with fellow justices later on, after oral arguments, they’re convinced that maybe they didn’t get it right in their eyes, maybe they were convinced strategically to move in another direction. So obviously as you get smaller courts, the likelihood of this is diminished. If you have a one-person trial court, there’s not going to be this discussion after the fact, but the more justices have, the more judges you have, the more likely, at least a possibility is that something occurs after the initial vote on the merits, but before the actual decision is released.
Badri Narasimhan:
So this is the end game. And as you can see, we kind of give a, which way is a certain justice leaning and we’ll walk back from there into how we got there. The way it all starts is we upload a new case and we can either point it to an internet source of the audio or we can upload the audio file, one way or the other. And once we do that, our system, and this is the AI part, and then the algorithmic part, the system generates a transcript of the audio and it identifies individual speakers and we split the audio into question and answer segments where a speaker is asking a question, a different speaker is giving a response and we then assign, hey, is it a justice or is it an attorney? And we help the system in the way that it says, hey, this particular speaker says, “counsel”, you decide whether it’s a justice or not, but the system thinks it is a justice. And this particular speaker says “your honor” and you decide whether that is the attorney or not, and so on and so forth.
And so we’re able to understand that there are X number of speakers, the speakers are most likely one type or the other, and then we create a visual plot. A visual plot shows, and this is where Adam was alluding to the tonal analysis. There are two components, “what is said” and “how was it said,” and the “what is said” inform certain parts of the metrics and “how was it said” informs other parts of the metrics and the “how was it said” also has some components that show turning points. So here is Justice Kagan interacting with one of the attorneys and Justice Kagan all the way over here. And as you can see Justice Kagan’s engagement, which in our case when the graph goes higher, it means she is less likely to favor that point.
And the easiest way I explain is when you talk to your own kids at home and you are very nice and you say, “I’m so happy you did that.” And now when they do something that you don’t agree with, you don’t say, “I’m so happy you spilled the milk all over the kitchen counter.” You say, “Why did you spill the milk on the kitchen counter?” And that’s really what we are doing except we are taking 100 samples a second and using that to inform our algorithm. And so we generate a transcript, we generate samples per second. These are the attorneys, the blues are the attorneys, that’s Solicitor General Prelogar and this is Hawley. And we are then able to say how did each one of them interact with each justice? And here is more of how the watch works behind the scenes. There are multiple parameters for each justice-attorney pair. And the composite of all of that is what generates a weighted index, which tells us favorability. Anything you would like to add, Adam?
Adam Feldman:
Yeah, two things to add. One is that it’s important to note that we’re looking at each justice or judge separately. So we’re not comparing the way that Thomas speaks to Justice Roberts, Chief Justice Roberts. We’re looking at a judge relative to themselves as they’re talking to both sides. So when we look at the slope of the justice, so how high were they speaking? And that’s more emotional to one side versus the other, we can actually look for justice at how they’re speaking. So we’re not interested in necessarily comparing the way that the justices are speaking because each of them has the words that they used and the tone that they use.
So it’s not going to be helpful to compare the type of audio from Chief Justice Roberts with Justice Alito. We’re looking at the justices individually. And one other thing that’s really interesting I think about our system is like Badri was noting with Justice Kagan how she was speaking differently to both sides, we can actually look at those individual moments. So if we pull down to the graph and we’re looking at Justice Kagan at one of the higher low points and we’re interested in actually knowing what’s going on there, we can click on it and it shows us with the text what’s being said at that moment and so-
Recording of Justice Kagan:
Where are we looking for that?
Adam Feldman:
So just in those little snippets, we can actually then see if what is on the page correlates to what we’re listening to. So we can see where there might be some change in a judge or justices feeling about a case. We can actually measure those specific instances. So while the analysis, and I think the aspect that most people are going to focus on is predicting outcomes, we can actually understand the interaction at a much deeper level by leveraging the ability to look at the individual segments of speech.
Kevin Skrzysowski:
I think this technology is absolutely incredible and I’m envisioning many different applications for this product. But let me ask you, what are some of the best use cases that you’re considering right now and who is your ideal target market for this?
Adam Feldman:
Well, so we’re very much in the exploratory phases. Having in mind I would say at least two segments, and Badri maybe wants to add to this, but my initial gut with this was that trial attorneys would be the ones who would benefit the most, because you’re settling at trial after certain motions are argued, where there’s audio. And so you can look at different segments of a case, weigh a judge’s feeling about it and you can even get more out of the audio most likely
than you’re going to get out of a text response in some kind of judgment from a judge. Because we can actually look at fine-tuned analysis, look at how much a justice or judge is feeling positively or negatively about a position in the case and one side relative to the other. So we’re looking rather than a binary yes or no at a spectrum of choice.
So that can inform settlement offers at a much more precise level, because you’re not only looking at the possibility of winning, but at the probability of winning. So you can actually fine-tune types of analysis on how much you should offer or take at a settlement. The other main interest that we were thinking of, we have in mind for this, has to do with financial stakeholders that can look at an oral argument and that still have room to maneuver after the argument is made, because we know we’re looking at a limited universe of people, at least for those who are actually litigating a case that can make use of some type of decision-making after oral arguments. So one might be financial stakeholders that are invested some way in the outcome of a case that have room to maneuver after the oral arguments. There’s a whole slew of other possibilities. Maybe Badri can talk to this about how attorneys might be able to use this as training also that we can actually use to show specific points in the oral argument that might be more valuable to understand.
Kevin Skrzysowski:
I think that’s super interesting. And I was going to say, what about the private equity and hedge fund markets who are in financial stakeholders in those companies whose business valuations or bottom line could be impacted by the decision that’s coming down the pike? That’s very interesting.
Adam Feldman:
Yeah, and just to bounce off of that, when I was doing this as a practicing lawyer and then a political scientist, most of the questions that people would ask of me knowing that I was in the analytics world had to do with those types of cases. There was one in particular where I worked with a hedge fund looking at a case where the fund was heavily invested in a specific sector. And so I was able to make a prediction and it wasn’t only based on oral arguments, it was based on past trend data that predicted the same kind of outcome. I got eight out of the nine votes correct on that. And so I assume that was used as a tool to make some decisions at a later phase in a case. So this was up at the Supreme Court level, it wasn’t at a trial court. So I know that there are investors that have used this type of analysis or predictive analytics in general. And so we’re really looking at folks who have similar interests and that want a higher level of predictive capacity that wasn’t available prior to 2024.
Kevin Skrzysowski:
I think those are all great markets, great markets for you. And this program goes out in our bi-monthly newsletter to our contact list, which is 10,000 people, which are mostly lawyers from AM Law 100 firms and investment banks and private equity and PortCos. So if there’s any members of the audience who are listening and they would like to contact you to discuss this, what would be the best way to get a hold of you?
Adam Feldman:
I think through email would probably be the easiest way. And it’s an easy email address to remember if you know my name, it’s Adam A-D-A-M @feldmannet.com. So F-E-L-D-M-A-N-N-E-T.com.
Kevin Skrzysowski:
That’s great. And I would also like to encourage the audience to go to your blog, Empirical SCOTUS. It’s a fantastic resource that you’ve put together.
Adam Feldman:
Thank you.
Kevin Skrzysowski:
Well anyways, gentlemen, we’re at about the 30-minute mark and I like to keep these programs within the commuter timeframe. That’s where we get most of our listening audience. But I just want to say Adam and Badri, a real true pleasure to have you on the program today. And I want to thank you for all the time you took discussing and demonstrating this innovative and transformative technology that you’ve developed.
Adam Feldman:
Thanks very much for having us.
Badri Narasimhan:
Thank you.
Kevin Skrzysowski:
And of course, as always, I’d like to thank the audience for listening. If you’d like to hear more, please be sure to follow us on Apple, Spotify, Stitcher, or anywhere you listen to your favorite podcasts. And if you’d like to learn more about the litigation, insurance, and finance products that the Certum Group provides, please visit our website at www.certumgroup.com and you can always reach out to me directly at [email protected]. Again, Adam and Badri, thank you for your time. Thank you to the audience and until next time.