What are the implications of NLP and AI for legal tech?

Welcome to LEGALTECH MATTERS, a Litera podcast dedicated to creating conversations about trends, technology and innovation for modern law firms and companies big and small.

00:00:14:18 - 00:00:27:23
Ari Kaplan
Welcome to Reinventing Legal. I'm Ari Kaplan, and I have the privilege today of speaking with Tonya Custis, the Director of AI research at Autodesk. Hi, Tonya, how are you?

00:00:28:05 - 00:00:30:01
Tonya Custis
I am great. Thanks for having me today.

00:00:30:09 - 00:00:36:18
Ari Kaplan
It's my privilege and I'm looking forward to this conversation. So, tell us about your background and your role at Autodesk.

00:00:37:07 - 00:01:05:21
Tonya Custis
I am the Director of Research at Autodesk, I lead a team of research scientists that do fundamental and applied research on AI and as it is relevant to Autodesk industries. Autodesk is a software company and we make computer aided design software for three industries architecture, engineering and construction, manufacturing and media and entertainment. So we make lots of different kinds of software, but the ones that people are most familiar with are AutoCAD and Maya.

So really, if you've ever ridden in a car or been in a building or played a videogame or watched a movie, you probably experience some of the things that are customers have designed with Autodesk software. So I'm coming up on just over three years at Autodesk. Before this, I was a senior research director at Thomson Reuters and I worked on AI features for Westlaw.

And before that I was a research scientist also at Thomson Reuters and at eBay and at Honeywell.

00:01:37:08 - 00:01:48:12
Ari Kaplan
Your major in college was music, but then you pursued graduate degrees in artificial intelligence and computational linguistics before earning your Ph.D. in linguistics. How does the music degree relate to the others?

00:01:48:20 - 00:02:02:02
Tonya Custis
It's absolutely related. Music, linguistics and computer science are all really just about recognizing and leveraging patterns. So there's a very common thread that goes through all three.

00:02:02:10 - 00:02:21:06
Ari Kaplan
It's funny because my daughter is amazing at recognizing patterns. She's fantastic at word games and calculations, and I'm always wondering if the future for her is in some job I've not even heard of or thought of yet. What type of artificial intelligence were you studying 20 years ago?

00:02:21:18 - 00:02:56:09
Tonya Custis
In grad school we were mostly doing things in robotics, things like control and planning, rule based systems. Also some perceptrons, which are really just one layered neural networks. Very simple compared to what we do now, and really just more classical like kind of statistical machine learning techniques for supervised learning. We have things we still do now. I think what most people don't really realize is, you know, the mass of AI hasn’t really fundamentally changed since the 1950s and 60.

It's a lot of linear algebra, but it's really just the scale that's changed. So what really has brought advances in AI has been the ability of a lot of digital data and also the compute power to train really large models on that data. You know, so previously, even as we had maybe more and more digital data, we did not have maybe enough compute power to train on the billions of parameters of models, the size that we're training on today.

00:03:29:05 - 00:03:36:09
Ari Kaplan
How would you distinguish data science, artificial intelligence, and natural language processing?

00:03:36:24 - 00:04:05:04
Tonya Custis
Artificial intelligence is using computers to do tasks and mimic behaviors that are typically thought of as being done by humans. To unpack that a little bit as more and more things can be done with computers. This definition kind of shifts like we often just think of AI as like the things cool things computers can't do yet, but there are plenty of really cool AI things computers can do now that we kind of take for granted, like speech recognition.

So typically now when we talk about AI, we're also talking about machine learning. So we use machine learning to do AI, although there's other ways to do. I like evolutionary algorithms or good example, but machine learning really is just a collection of statistical techniques that learn from and recognize patterns in large amounts of data. So then natural language processing and NLP, it's a branch of AI.

So we use machine learning to train models to do linguistic tasks, right? So things like question answering, summarizing lots of documents or generating text. There are other branches of AI that also mimic other human capabilities like computer vision and speech recognition. So my career has been primarily doing research in NLP. And so but generally AI research scientists do research on novel algorithms and machine learning techniques that contribute to the science of AI through rigorous experimentation.

So we published papers on the results, and then these techniques make their way into commoditized software tools that other people can use. So other people, like data scientists, use these tools. So data science is a discipline that uses kind of commodities. AI and machine learning software tools and libraries to do data analysis and predictive analytics. They're training machine learning models know about the data that they're training on, and they're using these tools, tree models to make predictions.

00:05:38:19 - 00:06:03:14
Ari Kaplan
I'm so fascinated by the music connection and the patterns and how they relate to all of this. Do you recommend that folks who are thinking about disciplines like music or others that involve patterns can consider how that application to this sort of future looking AI or NLP or data science potential is available to them?

00:06:04:11 - 00:06:23:16
Tonya Custis
I don't know that I had any sort of master plan at all. I think maybe you're giving me too much credit here. So but I do think I mean, for kids who are interested in math or music or vice versa, like that's a really interesting kind of connection. And I think there is a lot to be said for that.

And to be honest, I feel like I'm not actually that unique. There are a lot of people that I have run into in my career who are musicians.

00:06:31:21 - 00:06:56:04
Ari Kaplan
I'm sure I'm just thinking about how cool this session is. This discussion is basically like music to AI, and I just think people are going to be fascinated by that. I don't think everyone thinks about that unless you're kind of in the space. So after earning your Ph.D., you mentioned spending you spent seven years at Thomson Reuters as a senior research scientist and then were at eBay and Honeywell in various roles before returning to Thomson Reuters.

Were the legal use cases at Thomson Reuters equally suited to natural language processing research as much as those at the more mainstream organizations at which you work.

00:07:06:11 - 00:07:46:11
Tonya Custis
Absolutely. So legal language, it's really, really interesting and it's also fun to work on because it's, you know, purposely very precise but also purposely ambiguous at the same time. So it is special in its own way. Right. But regardless, the domain you're working in. So if it's e-commerce or manufacturing or legal or architecture, whatever, every specialized domain has a specialized vocabulary, often has specialized styles that works in there's knowledge, there's different kinds of data that need to be represented and encoded in the AI models that are being trained.

So kind of what all these specialized domains having come in is that general purpose models trained on the Internet. Say like ChatGPT, really don't usually perform up to the standards needed to sell software in a specialized domain like legal or architecture. So I used to say this a lot at Thomson Reuters, right? The research opportunity is in kind of getting the domain specific models to perform well enough, you know, to charge people to use.

Right. So, this is the other thing. Like a lot of AI models, like we work on them in the accuracy is like 40%, you know, that is not good enough to charge a customer for it, right? So it's really about like in these domains, like TR and Autodesk, like how do we improve existing and create new technologies that can perform well on the different kinds of data in the different domains.

Right? So it's very specialized, but at the same time, it's kind of similar from domain to domain, I mean, just doing the same thing with different data. So at Thomson Reuters, the challenge was legal data, because legal language is challenging. At Autodesk, the challenge was is 2D and 3D geometric data.

00:09:02:14 - 00:09:20:20
Ari Kaplan
It's so I'm sure you're talking about this. How long do you think it will be before there are kind of universal commercial applications of AI? We see this now. There's a lot of discussion about being free, but then there's a premium. How long do you think before it's just embedded in everything? And we are at this point where it's fully commercialized?

00:09:21:17 - 00:09:40:20
Tonya Custis
Well, I mean, I think it's embedded it in more things than you think about, to be honest. If you think about your, you know, autocomplete on your phone, it's you know, it's making decisions behind the scenes and a lot of things that you probably aren't even thinking about, even just like things you take for granted, like recommendations on Netflix, right?
So it's already really embedded in your everyday life. But I think we're a long way off from kind of a general purpose solution that can be used across different domains. It's just to now again, there's like value to the customer aspect, but then to there's just, you know, it's not magic. Think about how many hours a person has to study to learn these domains.

You can't. It's not magic. I think we're a ways off.

00:10:11:06 - 00:10:17:18
Ari Kaplan
How does the work of the AI Lab research team at Autodesk differ from your prior leadership roles?

00:10:18:13 - 00:10:45:03
Tonya Custis
Honestly, it's not too different. The concerns and responsibilities of AI researchers are pretty standard across the different groups and like industrial labs at different companies. A lot of research scientists career is defined by their publication record in OUT in the AI research community. So since that's external to any one company, there's kind of a lot of consistency actually from company to company.

So, I think it's not too different. Obviously like different companies, different cultures and whatever. But I do think one thing that is different in Autodesk is my AI research team is only one of several teams inside of a bigger Autodesk research, whereas kind of in most tech companies, most software companies research kind of implicitly just means AI research. So like if you think of like research at Google, it's like all AI researchers, right?

And at TR is all AI research. So what's cool at Autodesk is there is other research groups that are working on things like they're more specific to Autodesk industries and like the future state of those industries. So Autodesk research or some weird stuff like lunar landers with NASA or turbo engines for Rolls-Royce or, you know, just weird kind of concept things that are really cool. And it's really interesting to get to work as part of a bigger team that's working on, you know, things that are a little outside of our research areas.

00:11:49:19 - 00:11:56:17
Ari Kaplan
How often is that entire global team together discussing the different projects and maybe the inner relationships between them?

00:11:57:13 - 00:12:08:07
Tonya Custis
Think the whole global team only together like once a quarter or something, but there's certainly a chain of command and a lot of cross-pollination that happens.

00:12:08:07 - 00:12:12:23
Ari Kaplan
What types of projects do you specifically work on and has that changed over the last few years?

00:12:13:20 - 00:12:40:02
Tonya Custis
My team does projects mostly centered on 2D and 3D geometry reasoning and understanding and multimodal generative modeling. So, in the past couple of years that, yeah, well one of course last couple years we've really shifted more towards the multi-modal generative modeling, right, which is just a fancy way of saying we're doing more research that trains large generative models on data of different modalities.

Okay, so data like text, like 2D and 3D data and you know, you train model and all these things together. So my team was actually the first lab to publish in the text to 3D generative space. So, as you can imagine, that's super-hot right now with all the text to whatever models happening. So we have models that take text as input and then the output is a rough 3D model.

So I mean, but ultimately we want to get to the point where we can say make a chair and it would produce a manufacturable 3D model that we could also use. And the semantics of geometry to manipulate. So, you could say something like make it rounder, make it square or make it taller, give it four legs instead, or give it five legs.

So, you know, the idea here is to make the tools easier to use so we can save time for designers and maybe give them some creative inspiration sometimes definitely not to replace them just to help them. One of the reasons I was so excited to come to Autodesk really was just this opportunity to combine what I know about natural language processing with geometry processing, and to be able to combine language and geometry to like inject this like semantics into geometry so people can refer to and use language to help them design things, right?

But almost just to translate their ideas into geometry. Right? Like language is the best thing we have to do that with. So yeah, it's a really exciting time to be in this space.

00:14:15:21 - 00:14:35:22
Ari Kaplan
I'm envisioning this sort of Tony Stark Iron Man discussion of make this shorter, make this longer, and harness this particular aspect of it. Do you see a future like that where you're actually giving those instructions and it's materializing as you're speaking in the same way that you see ChatGPT writing a white paper.

00:14:36:18 - 00:14:55:22
Tonya Custis
Like right now we can do it at just the model we get is kind of rough and it's not manufacturable. So it would work for the media and entertainment business like you can do it for. You're just rendering it out on a screen that's a lot less precise and you know, building a building or a bridge or manufacturing an actual thing.

So, yeah, I mean, we're not too far off, honestly. I think there is just so much happening in that space right now. Like every day there's more papers to read. Every day we're behind that, but it's really exciting.

00:15:09:07 - 00:15:12:22
Ari Kaplan
How did you manage your team and this workflow during the pandemic?

00:15:13:04 - 00:15:42:15
Tonya Custis
Honestly, we're already pretty geographically distributed, so we're in seven different locations in five different countries. So we are mostly Zoom operations. And so research workflow didn't really differ too much other than conferences were remote instead of in-person, you know, and we didn't get to be together in person maybe as often as we would have otherwise, you know, periodically who all gather in one place or something.

But yeah, we're pretty spread out anyway, so I honestly didn't does too much.

00:15:47:18 - 00:16:03:15
Ari Kaplan
We talked about your experience with music and this idea of pattern recognition, and then you also mentioned that the technology is changing so rapidly. What skills do you find are most important for new hires in the AI lab at Autodesk?

00:16:03:15 - 00:16:28:04
Tonya Custis
Well, I mean, definitely expertise and experience and machine learning, but assuming that's covered, the most important thing I look for is somebody who is a creative problem solver and someone who is curious. So I have found that the people who are most successful because it changes so fast, because mostly the job is just like keeping up with stuff and learning new stuff every day.

The most successful people in this role are people who can take what they know from their experience and creatively apply it to a new problem and who are curious about how does it work? How can I you know? So yeah, I definitely am creative problem solving and curiosity.

00:16:46:20 - 00:16:59:10
Ari Kaplan
Speaking of that, there are many more roles related to machine learning data science, innovation and knowledge management in legal today. What does that indicate about the practice of law?

00:17:00:06 - 00:17:39:04
Tonya Custis
To me, what it says is it really speaks to how legal is and has been data centric domain. So as modern software kind of shifts from this like code centric paradigm to a data centric one, I think we'll see more and more really cool things in legal tech. Maybe relatedly, this is why legal tech was a bit slow to take off or it was kind of clunky and awkward, right, for a while relative to other domains because like that code centric software paradigm really could not accommodate the complexities of an industry that is so dependent on data.

Ingesting data and generating data. That's the currency. And so really I think that's what it says to me is that almost like the software development paradigm is and machine learning are like catching up to legal in ways that maybe before they weren't.

00:17:59:14 - 00:18:07:15
Ari Kaplan
You see all of the recent attention that has been focused on ChatGPT making AI more familiar or more confusing.

00:18:08:12 - 00:18:33:08
Tonya Custis
ChatGPT and other generative models, you know, they're making some pretty sophisticated AI available to a lot of people, and it is super fun to play with. I mean, honestly, it's the thing I have been maybe working towards my whole career, but not everyone really understands or is maybe thought through the fact that language isn't the same as thought.

Right. So ChatGPT and other large language models are optimized to mimic the structure of human language, right? Not the content. So that can be really confusing, I think, to people. You know, the models are using patterns that they learn from ingesting huge amounts of data, right? The Internet. And they use what they've learned to mimic the structure, to mimic the patterns of human language.

But, you know, there's no meaning, there's no intention, there's no knowledge, there's definitely no fact checking. And so really it's like exquisitely crafted word salad. It's just it's psychologically confusing to hear something that sounds like it comes from like a very educated person, but it's basically a bunch of factual nonsense half the time.

00:19:25:23 - 00:19:50:11
Ari Kaplan
I was looking at your bio, you have several patents. One of them is systems and methods for generating contextually and conversationally correct response to a query which relates directly to what we were just talking about another information retrieval systems, methods and software with concept beat searching and ranking, can you share some details about what was involved in that effort which took place long before this was introduced?

00:19:51:06 - 00:20:18:07
Tonya Custis
Like publishing papers, patents are just part of the job when you're doing AI research right, regardless of the company, they all want to protect their IP. So these patents just come from developing novel machine learning techniques that are applied to solve problems in new ways. So with the contextually and conversationally correct response to query, right, that actually is, I think was for the west search plus question answering.

Right. So it's just when asked a question that the answer we generate would sound like a person answered it. So it's like in discourse rule, we all follow discourse rules. You and I are following certain rules about how we have a conversation. And, you know, when somebody does something weird, you know, it's weird, right? So this patent is about, like, not being weird, answering it in a way that a person finds, like, flows from that conversation.

Right? And, you know, I would say the patent I'm probably most proud of is the concept searching one. So as an early example of using kind of the statistical distribution of words as a proxy for semantics. So you can think of it like we do this, like if you don't know what a word is, you figure it out like what it means from the words around it.

And so that is honestly how all large language models like GPT three and how GPT works today, right? It's a lot of counting. And it's based on what the statistical distribution is, what words are next to what other words. Right. So, yeah, so that one was a long time ago. We couldn't did not have the computer power to fire that up to the levels of today.

But the concept was the same.

00:21:32:03 - 00:21:47:16
Ari Kaplan
I'm having so much fun in this conversation. I'm just amazed at how relevant your experience over the course of your career is to what is existing now, what people are so interested in what's coming. Has your career evolved in the way you envisioned it back in the late nineties when you started college?

00:21:48:12 - 00:22:14:06
Tonya Custis
Let's be honest, I was a music major. I don't think when I started college I had any career in mind or anything. I've been really fortunate to kind of be able to have studied the things I loved, the things I thought were really interesting, and then to have been able to combine them in a way that I have been able to have a career where I can kind of continue to do research and continue to solve interesting problems.

My uncle asked me when I was in grad school, like linguistics and computer science, what are you going to do with that? And I'm just like, I'm sure I'll find something, you know? And I think that was Google was kind of starting like there were options that were coming out. But no, I mean, and really it came from a thing where I was really interested in how do children learn language.

So I was interested in how can we teach computers to understand language.

00:22:43:11 - 00:23:07:17
Ari Kaplan
When I was in college, I spent several years working on an intergenerational literacy project, and I was so lucky to work with experts in that field of education. And it was incredible to try to understand how language is processed. And I actually always say to my kids, when I'm listening to you, I always say to my kids who are amazing, I always say, you know, I don't know what you're going to do, but it's certainly possible that you will do something we haven't even heard of yet.

Like, I don't even I can't even anticipate based on it. I just it's so interesting here about your career and your background. This program is called Reinventing Legal. How has your work ultimately impacted that objective?

00:23:22:08 - 00:23:47:14
Tonya Custis
I'm really proud of the work that I did when I was at Thomson Reuters. I think in particular the West Search Plus question answering feature. The key site implied over rulings feature other stuff like before that I had just done web search algorithms and I did a lot of non-English search stuff for Westlaw. And on the concept search stuff we discussed and you know, the group there, the R&D group there, it started in the nineties.

I mean, it's just it's an has a really, really interesting and extraordinary history and track record. I feel really lucky to have started my career there. I think that was amazing and it was very nurturing career wise and I think I was also really fortunate to get to return there later to be a research director. So I mean, it's a really special group of people and they have a really special connection or working relationship with the Westlaw product team.

And this is something I haven't seen in any other company. And I do really credit relationship to the success they have had churning out those features and the quality of those features.

00:24:29:21 - 00:24:41:00
Ari Kaplan
Well, I'm Ari Kaplan and have been honored for this edition of Reinventing Legal to speak with Tonya Cust Custis, the Director of AI research at Autodesk. Tony, thank you so very much.

00:24:41:07 - 00:24:42:16
Tonya Custis
Thank you. This is super fun.

00:24:43:08 - 00:24:49:00
Thank you for listening to LEGALTECH MATTERS. Be sure to subscribe wherever you get your podcasts.