Civil Law

In this episode of The Geek in Review, we welcome back Pablo Arredondo, VP of CoCounsel at Thomson Reuters, along with Joel Hron, the company’s CTO. The conversation centers on the recent release of ChatGPT-5 and the rise of “reasoning models” that go beyond traditional language models’ limitations. Pablo reflects on his years of tracking neural net progress in the legal field, from escaping “keyword prison” to the current ability of AI to handle complex, multi-step legal reasoning. He describes scenarios where entire litigation records could be processed to map out strategies for summary judgment motions, calling it a transformative step toward what he sees as “celestial legal products.”

Joel brings an engineering perspective, comparing the legal sector’s AI trajectory to the rapid advancements in AI developer tools. He notes that these tools have historically amplified the skills of top performers rather than leveling the playing field. Applied to law, he believes AI will free lawyers from rote work and allow them to focus on higher-value decisions and strategy. The discussion shifts to Deep Research, Thomson Reuters’ latest enhancement for CoCounsel, which leverages reasoning models in combination with domain-specific tools like KeyCite to follow “breadcrumb trails” through case law with greater accuracy and transparency.

The trio explores the growing importance of transparency and verification in AI-driven research. Joel explains how Deep Research provides real-time visibility into an AI’s reasoning path, highlights potentially hallucinated citations, and integrates verification tools to cross-check references against authoritative databases. Pablo adds historical and philosophical perspective, likening hallucinations to a tiger “going tiger,” stressing that while the risk cannot be eliminated, the technology already catches a significant number of human errors. Both agree that AI tools must be accompanied by human oversight and well-designed workflows to build trust in their output.

The conversation also delves into the challenges of guardrails and governance in AI. Joel describes the balance between constraining AI for accuracy and keeping it flexible enough to handle diverse user needs. He introduces the concept of varying the “leash length” on AI agency depending on the task—shorter for structured workflows, longer for open-ended research. Pablo challenges the legal information community to break down silos between disciplines like eDiscovery, research, and litigation, envisioning a unified information ecosystem that AI could navigate seamlessly.

Looking to the future, Joel predicts that the adoption of AI agents will reshape organizational talent strategies, elevating the importance of those who excel at complex decision-making. Pablo proposes “ambient AI” as the next frontier—intelligent systems that unobtrusively monitor legal work, flagging potential issues instantly, much like a spellchecker. Both caution that certain legal tasks, especially in judicial opinion drafting, warrant careful consideration before fully integrating AI. The episode closes with practical insights on staying current, from following AI researchers on social platforms to reading technical blogs and academic papers, underscoring the need for informed engagement in this rapidly evolving space.

Listen on mobile platforms:  ⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠Apple Podcasts⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠ |  ⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠Spotify⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠ | ⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠YouTube⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠

[Special Thanks to ⁠Legal Technology Hub⁠ for their sponsoring this episode.]

 

Blue Sky: ⁠⁠@geeklawblog.com⁠⁠ ⁠⁠@marlgeb⁠⁠
⁠⁠⁠⁠⁠Email: geekinreviewpodcast@gmail.com
Music: ⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠Jerry David DeCicca⁠⁠⁠⁠⁠⁠⁠⁠⁠

Guest’s Go-To Resources:

• François Chollet: Balanced AI insights.
  x.com/fchollet
  fchollet.com
  
• Jason Wei (OpenAI): Reinforcement learning updates.
  x.com/jason_d_we
  openai.com/blog
• Geoffrey Hinton: AI research insights.
  x.com/geoffreyhinton
• Richard Sutton: Reinforcement learning and philosophical takes.
  incompleteideas.net
  Reinforcement Learning: An Introduction (Book):
  http://incompleteideas.net/book/RLbook2020.pdf
  Seminal work by Turing Award winner.
• University of Alberta Lab: https://rlai-lab.github.io
  Current research on scalable AI methods.

Blue Sky: ⁠⁠@geeklawblog.com⁠⁠ ⁠⁠@marlgeb⁠⁠
⁠⁠⁠⁠⁠Email: geekinreviewpodcast@gmail.com
Music: ⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠Jerry David DeCicca⁠⁠⁠⁠⁠⁠⁠⁠⁠

Transcript

Nikki Shaver (00:00)
Hi everyone, it’s Nikki Shaver here with Legal Tech Hub. Little bit of a change of scenery than what you’re used to. Greg and Marlene have asked me to provide a little bit of an overview of Iltacon, where I’ve just come back from.

few takeaways for you One, there bit of panic in the market around AI vendors as we see them jostle for position, new vendors come out, so much funding. A bit of insight from one of the bankers is this bubble is not going to burst anytime soon. We are seeing more money than ever in the market and it’s justifiable. The TAM

the total addressable market of legal is bigger than what a lot of outside funders anticipated. So we should expect to see continuing funding rounds in the legal for the next few years at least. Another trend we saw at ILTA was discussion of AI teams. A lot of firms grappling with how to handle AI and actually upskilling internally, but also hiring people specifically.

sometimes sitting within IT, sometimes within innovation, and sometimes creating an entirely new department, spanning the entire firm, sitting independently. The third thing that we really saw this year is a real interest in potentially the rise of new platforms in the market, where previously you really thought of your core platform as Outlook and maybe…

a DMS system where you have the predominant documents, your work product sitting in the firm. Now, maybe we’re thinking about something else as being the core system where you still have your data repositories, but they’re pushing into the system where lawyers will do their work. So there’s a little bit of a notion that things might switch up quite dramatically. And the word on the street from people at ILTA that they expect things to shift quite dramatically.

both at law firms where there’s the potential for pricing changes, a lot of momentum towards looking at the value of legal work, not by the billable hour, although the billable hour will still be essential for calculating alternative fee arrangements, but more about how do you really look at individual workflows and what makes sense from a pricing perspective for those workflows.

based on how much the efficiency gains may be in legal. We’ve also seen a shift from AI being regarded as a pure product to gain an efficiency tool to something that can provide a far higher ROI. So actually greater value from your lawyers, from the client perspective, higher quality output, access to better data, and so on.

then on the vendor side, again, we really think there is a potential shift in the market coming where the players we’ve really seen as mature and enduring may not necessarily be the core vendors in the industry anymore and a shift towards some of the newer players who have entered only recently that may be those enduring platforms to come. So a very interesting ILTACon

and we’ll be writing about that on LegalTechnologyHub.com to find out more. Over to you, Greg and Marlene.

Marlene Gebauer (03:34)
Welcome to The Geek in Review, the podcast focused on innovative and creative ideas in the legal industry. I’m Marlene Gabauer.

Greg Lambert (03:40)
and I’m Greg Lambert.

Marlene Gebauer (03:42)
We are really excited this week to bring back one of our OG guests, Pablo Arredondo, VP of CoCounsel Thomson Reuters and Pablo, it is great to have you back on the show again. think you’re almost to the smoking jacket.

Pablo (03:54)
I’m heading there. Thank you guys. Great to be back.

Marlene Gebauer (03:55)
It’s a goal.

Greg Lambert (03:58)
Yeah, I

think like three or four, although Pablo would look good in a Tierra too, so let’s not typecast him.

Marlene Gebauer (04:02)
It’s true. Maybe,

Pablo (04:03)
Okay, well

we’re gonna, it’s a different show I think Greg, we’re gonna try all that out, flag that for later.

Marlene Gebauer (04:03)
you know, it’s true. He can choose. He can choose.

Joel Hron (04:07)
Yeah.

Marlene Gebauer (04:09)
And, and,

and joining Pablo is Joel Hron CTO at Thomson Reuters. Joel heads up the product engineering and AI R&D at TR. Joel, thank you for joining us as well.

Joel Hron (04:19)
Thank you for having me. Look forward to it.

Greg Lambert (04:21)
All right, so Pablo, ⁓ you reached out to us last week and suggested that we jump on a call to talk about the OpenAI’s rollout of ChatGPT-5. So let’s just jump right into the middle of it and tell us are your thoughts on the latest model from OpenAI and ⁓ what kind of impacts it’s gonna have on legal.

Pablo (04:43)
I’m so in your guys’ debt because you’ve allowed me to have these little video postcards from this journey of neural nets starting in 2018 lamenting, how come it’s doing all this great stuff for everything but language, right? Right before BERT came, right? Was this sort of just like everything but language seems to be going great.

And then BERT coming and you guys know how agitated I got about BERT. I was so excited because we were freed from the tyranny of the keyword prison. Right now we could do this sort of beginning of language. And then I think the next time, we can now we had a Darth from Ford and Evan, right? It was just kind of blasting with GPT4 And now, and then one I think that ⁓ would sort of maybe check. We won’t even worry about too much about that one, although it may have angered some Yale Law librarians and I would like to apologize to them.

invented and to thank them for the polemic that I was actually quite useful in places. And now though I think the big news is what’s happened since and GPT-5 is an embodiment of this but really it’s the advent of these reasoning models. And so I don’t know what to say guys, it’s like I don’t want to keep getting very excited about this stuff but the last one

They leveraged like everything ever written and pulled somehow out the statistical power of that and used it to do things like pass the bar exam or do doc review. But right. And then someone saw that and was like, what a stupid model that is. Let’s have it pursue correct math. Like, let’s have it want to. And this is reinforcement learning. They used to call it like hedonistic learning. Like the thing desires to be right at the math problem.

and then give it this countless ability to do trial and error and further tweaking. And so in that, it almost frees itself from the tyranny of the human word, right? If you follow, like I’m taking this from its perspective, it’s no longer just bound to how we’ve written about things. It’s just trying to do whatever gets it to the right place for the coding problem or the math problem. And that jump.

01 was I think the first release of it, now 03, now present in all of the state of the art models. That’s the thing to write home about. I hope I bothered you then, Greg, until we got to talk about it then. If not, well, sorry for the delay, you guys. have, you know, it’s been… So, okay, so where do we find ourselves in? So, well, you know, 10 years ago I was like, guys, we’re gonna drag the brief in and watch it use the citation graph to go find a case.

Greg Lambert (06:53)
You

Marlene Gebauer (06:55)
busy.

Pablo (07:06)
What we’re seeing with the reasoning models and you know, they can see this preview data suite in ILTA I think if they’ll show, know, is let’s put the entire litigation record in and its most fulsome state, the depositions, the expert reports, all of the requests for admissions responses, all, every thing that encodes fact and then armed with nothing but the complaint to ask the system to now go and map element by element with granularity, how do I win a summary judgment motion?

that’s the leap enabled by reasoning models that I don’t think you have from non reasoning models. And so unfortunately, I have to stay very excited and very optimistic. And again, put it out that optimism is born perhaps of my background being everything works out for me or something like that. you know, terms like there’s social like reasons why maybe I’m so happy about everything. But I hope objectively to convince you guys that putting all that aside, our system is well suited to benefit

from this ability of these models to backtrack and reflect, right, and kind of do it. gives us, in some ways, I think the final piece we need to build now, like, what will seem like celestial legal products. All right. That’s… Thank you, guys.

Greg Lambert (08:15)
Okay, well, we’ll bring you back on in 18

months and you can talk about how overly optimistic you were on this one and it is definitely not stopping. know, Pablo, you hit on something that I wanted to kind of dig a little deeper on and that was, you take the complaint and then you do a list of issues and causes of action.

Marlene Gebauer (08:18)
Hehehehehe

Pablo (08:21)
I’m sorry, I wanted to stop. I wanted to stop, but you can’t, yeah, I mean, okay.

Marlene Gebauer (08:22)
you ⁓

Greg Lambert (08:43)
and so forth. Do you think this will make for better lawyering or how do you think the lawyer is going to make themselves better by using this type of technology rather than just being lazy?

Marlene Gebauer (08:59)
And how will they change?

Pablo (08:59)
Right.

So there are some aspects of law that are uniquely and deeply human that go back to like, you know, long standing traditions, even of our ancestors, as, you know, customs recorded by Caesar when he went to the end of it. There are some aspects of law that should remain fiercely human. And then there’s discovery for relevant documents, Greg.

Marlene Gebauer (09:20)
You

Greg Lambert (09:21)
Yeah.

Pablo (09:23)
There’s

given this complaint, what are some good search queries to run? And this system.

leveraging everything ever written and you know there were copyrighted things thrown in there I mean there were some you could be mad about you know but whatever it was it got everything written and then came the trillion eons of meditation on the Pythagorean theorem until it organically learned self somewhere along the way this thing is the most formidable turning of a complaint into a bunch of legal research right

And what I want to say, I envision a system where we let it’s an and always an end. I don’t want anyone forfeiting anything right now.

But before we get into all of that, and I’ve spent years litigating, and I will say, if I take one issue with our good friends at Yale’s polemic at the legal tech community, the idea that we don’t sort of know about the indeterminacy of law, like any time you’ve done a motion to compel you, you can’t spend a second actually litigating without that being beaten into you very thoroughly. But that entire process, I think, right now is so slow, inefficient, prone to misconduct. think everything bad that can happen seems to happen in that, from the standpoint

And so what I would say is from the judges standpoint, after all, what are these scales of justice meant? But to receive the collective, here’s what I found that’s relevant and weaved it into my story. I think we can transform that, but I think it needs to happen at the level of like the federal rules of civil procedure. I’m not doing trainings on this one, Greg, I’m sorry. I’m it’s not, I’m sorry. We tried it, we tried the nice way and it’s great.

There’s also I think a lot of incentives for very good attorneys who’d prefer to have the imbalance in quality and let their ability prevent, right? Like there’s all these other reasons why maybe, right? So anyway,

Joel Hron (11:05)
I think an answer that gets to the same point, but like from a whole different perspective, given that, that I’m not a lawyer and I know like a fraction of a percent of what Pablo knows about law. If I look at like engineering dev tools, which I would argue have outpaced other AI tools and any other.

quite frankly, just because the models were better at this skill earlier, the language is better structured, it’s testable, there are many reasons for this, but AI dev tools have just been on a rocket ship of capability. And if I look at the impact of those tools on my engineering workforce and a lot of the peers I talk to, it’s not like it’s this sort bar leveling.

It’s not like it’s raising everybody up to this mean level of proficiency and capability. It makes the best engineers even better at their work. And the reason for that, I believe, is that it actually amplifies the hardest parts of the job. So in other words, all of the easy parts of the job that are about doing boilerplate work or this kind of thing are taken out of the equation. They’re expedited exponentially.

And more of your job is making the hard decisions, the hard steers, the hard redirects of the AI model to make the sort of harder engineering decisions around architecture and things like that. And that’s why I think the best engineers are even better. And I think the same will be true in many industries, right? It will amplify the hardest parts of the job, the most human parts of the job, as Pablo said, that require the deepest levels of sort of judgment and decision-making.

And I think that at the end, that’s a good thing for everybody. I think it improves the quality of engineering, it improves the quality of legal practice, know, and, and, and. So I think if we can look at that as sort of like a signal of where this may go, I think that could be it.

Marlene Gebauer (13:01)
Joel’s speaking about ⁓ impact. as, as you lead the R&D on an AI at Thomson Reuters, I know you have to keep an eye out for these changes in the AI models and how that affects the TR product CoCounsel and how that reacts and answers questions and prompts. Now, for an example, you recently released deep research for CoCounsel and can you tell us about those new capabilities that deep research gives us?

how the shifting landscape impacts a tool like that and the different models that you use.

Joel Hron (13:34)
Yeah, it’s probably the, I think one of the most pivotal things we’ve done since the beginning of this evolution of generative AI, quite honestly, I to me, this release is maybe more profound than anything that we’ve done before. And I think,

Pablo talked a little bit about reasoning models and made some big promises about what’s going to come in 18 months on the back of that. But what I would say is, if you look at the earliest version of these generative AI models, they had this world knowledge that was super impressive. They could answer a question about any topic in the world and it was quite amazing how general they were. I think if you look at the latest versions of these,

they still have this sort of world knowledge that underpins them, but they’ve actually narrowed in their capability and they’ve narrowed around things like reasoning and coding and math problems and logic. And what that signals is that these models aren’t just like a one shot answer generator to any question. They are, they’re like an operating system.

they are a tool with which through a prompt you program it to go do a task, right? And what that relies on, just like any other operating system, is other tools that it uses to go do the job. And I think that’s really the foundation of deep research. And I think that’s one of the main benefits we have of working on both sides of the model and the content, is that we are able to work with these reasoning models and understand

how they interpret information, how they plan a sequence of activities, how they execute tools, how they react and re-adapt their plans to new information. And we’re able to build tools, we’re able to build content sets, we’re able to access to tools like KeyCite that the agent knows how to use. And we’re able to build those tools in a way that makes it easier for the agent to use those things. It makes it easier for the agent to…

get an output from KeyCite and say, I see this breadcrumb. This is how I’m going to go follow that breadcrumb trail. And I think working on both sides of it, we’ve spent as much time on designing the tools in a way that the agents can use as we’ve spent working on the models themselves. And I think that sort symbiotic relationship between the two is really important. And when you do it well, I think yields tremendous.

results and what we see with deep research I think is really like I said a profound shift in these models being able to certainly write good plans in the same way a legal researcher would write these plans but also to be able to leverage these tools like KeyCite or the Key Number system to go follow a bunch of breadcrumb trails and see where they lead and if they lead to dead ends back up and and replan.

And I think the speed at which that happens is just a tremendous shift to how we think about this part of the business and super exciting for what’s to come with it as well.

Greg Lambert (16:38)
So on that Joel with the with with being able to follow those breadcrumbs, I got ⁓ a couple of questions that popped in my head on that was ⁓ one as the researcher. Am I going to be able to see the path that the that the AI is taken so that I understand you know where it found that breadcrumb and where it went from there and then also.

You know, we can’t talk AI and legal research without saying, know, how does the tool like deep research, quantify the hallucination risk and, you know, so that we’re not following something just because it’s now it’s made up something that sounds good.

Joel Hron (17:18)
Yeah, yeah, so I’d love for you, there’s demos of this going on ⁓ at ILTA, but we’ll also post demos online. I’d definitely encourage you guys to go take a look. You could see some of those things in real life about how that’s done. So the TLDR is absolutely yes to those questions. think the longer form answer,

Andrej Karpathy gave a YC talk a week or two back, which I thought was really good, worth a listen. And one of the topics he talked about in that was the user experiences of software applications being optimized for the generation verification loop. So if you think about a user experience that was designed like previously, right, it was all about

putting buttons in places that allow the user to sort of take actions when they needed to and to sort of make that feel like a seamless process of action. The user experiences of the future are all about verification of what AI generates. And so I think the better and more seamless you make that experience and the more often and the quicker you can allow humans to say, yeah, I agree with that path or no, I don’t go re-steer this way.

I think the better experience it is overall, the more trust the user at the end of the day has about the answer they’re getting. if you look at the demos that you can pull up online, you’ll see that in real time, you’re able to see the trajectories that the agent is following, what cases it’s looking at, why it’s looking at those cases, what it’s doing next and why. And also at the end of the analysis, you’re able to effectively

read through the thought process of the model. Like here’s what it did and why it did along the way, if you want. And then within the final report itself, to your point on hallucinations, we include really clear visual signals for users around what might be a hallucination or not. So for instance, if we reference a case.

we’re gonna go try to resolve that case against a real case that exists in the Westlaw database. And if we do, there’s gonna be a bright blue hyperlink to that case in Westlaw. If we don’t, there’s no deep link back to a case that doesn’t exist, right? And so these are like clear visual signals that we try to build into the product. We put key site flags in those answers in the same way you might see them in Westlaw. And so I think these visual signals,

are intended to really drive the user towards understanding what can I trust, what can I not trust, what do I need to verify, what do I not. And that’s all part about speeding up this generation verification loop in the user experience.

Greg Lambert (19:56)
good.

Pablo (19:55)
if I can just

add one thing on that, that the the key site flags themselves, citator flags were a technology of sorts that if used properly could speed things up and you would be more efficient. But if over relied on, I think this is Paul Callister who looked at this, right? The folks that just see the red flag and don’t go any further. Something that now the AI deep research tool is like, no, no, no, it was only overruled. So it’s interesting that it sort of overcomes other maybe misuses of it. But, ⁓ you know, make no mistake, the

Marlene Gebauer (20:17)
Just one part, yeah. ⁓

Pablo (20:24)
⁓ The word hallucination and this is also from the great Andres Carpatti like in some ways is the wrong word Because it implies that one thing’s happening and then it kind of loses its senses. It goes into something different If I can borrow from the great Chris Rock that that tiger didn’t go crazy that tiger went tiger, right? Like these systems even the ones that have gone through the sublime, you know like learn self-reflect whatever that is are still just guessing the next word is

Greg Lambert (20:35)
All right.

tiger went tiger.

Pablo (20:50)
That’s the system, right? Without any world model or system around it. That said, they’re doing it with increasing accuracy. And with strict oversight, there’s no question we can use it. One of the best in the, know, Andrew Ng, the professor at Stanford, one of the great AI minds.

I said, you know, AI is the new electricity. And I was thinking about that reading. There’s a book called The Last Days of Night about a young Paul Cravath who had just hung his shingle and was going to meet Thomas Edison to talk about the light bulb litigation. And the first sentence is like, as he walked to meet Thomas Edison, somebody was lit on fire in the sky above him because they were trying to get the street lamps to work. Right. And then, people. And so you, I think there are some lawyers that would have preferred some form of electric shock to what the judges put them through when they relied on this.

I think it’s unfortunate that we have that kind of misuse happening, but we really can’t let it deter us, I think, from thinking about how do we use it correctly. And I think that that’s important.

Greg Lambert (21:46)
Well,

Joel, let me ask you this because this is a question that a lot of my librarian friends have been asking. As we see almost weekly, sometimes daily, someone, whether it’s a pro se litigant or even someone at a big law firm that goes out and uses a chat GPT for legal research doesn’t

check all the citations, doesn’t read the cases and uses it. And then we even saw a judge last week, they had to retract a decision because they didn’t catch the hallucination. So the question that a lot of us are asking is why doesn’t a tool like KeyCite instantly pick up when a case is hallucinated either.

You know, it’s a citation that doesn’t exist or it’s a citation that’s citing to the wrong case or it’s a real case but it’s making up the language that is speaking in the case. I mean, as the engineer, what’s kind of the difficulties of being able to quickly identify when someone in a brief filed with a court is not.

Pablo (22:57)
Joel, they want that

sooner than 18 months is what he’s telling you right now. He doesn’t want any delay on that one, all right?

Greg Lambert (23:00)
I

Joel Hron (23:01)
Yeah.

Marlene Gebauer (23:01)
Hahaha

Greg Lambert (23:04)
won it yesterday.

Joel Hron (23:06)
think one of the interesting things about deep research is that, you know, I talked about tools and like, you know, for deep research, like the ability to go execute Westlaw searches or the ability to go retrieve cases, all these things are obviously tools. The ability to use KeyCite is a tool for deep research.

But we’ve also built things like exactly what you just said, which is like verification tools. Like when we cite a case, we want to go verify, is what this answer says, what this case says. So those are tools that the agent has available to it, the deep research agent has available to it to mitigate the risk of hallucinations. I will never say to you though that it will eliminate the risk of hallucinations. I think that’s an impossibility in

the way that these models work. The fact of the matter is that they are stochastic and that they can make mistakes on these kind of things, even though we’ve built a tool to do that. We have no guarantee in 100 % of the cases that that tool will get called in the right way at the right time. And so I think that’s the fragility in it. We spend a tremendous amount of time with…

our in-house legal experts and editors, putting this system through validation and optimizing for those kinds of errors and mitigating it as much as we can through how we change the architecture, how we change the prompts, how we change the configuration of the tools. But to say it will go to zero with infinite zeros in the decimal place.

I think it’s just something that I think anybody can actually say at this point in time, just based on how the models themselves work. we do build.

Pablo (24:47)
Meanwhile, how many does it

catch? How many human errors is it catching at the same time that it’s producing this, right? Like we have an errata, we have an actual Latin word that we brought out just for like, I screwed up.

The two mandatory CLE topics in California, think, are substance abuse and bias, which think about what that like what can we, you from that. you don’t have this like perfect baseline. And we’re starting to get to a point where we’re measuring each other like, ⁓ we got to this and we got to that. like then two things, I think that work like are that hit us that we have to do it. Where are humans and then also where it’s just Anthropic out of the box. Right. Those are these two lines that I think need to be talked about more than they are.

Joel Hron (25:25)
I agree with that, Pablo. I mean, I think the human baseline is not a perfect answer either, as you say. And so I think the use of AI doesn’t like obviate any of the things that a lawyer would have had to do responsibly previously. You know, like the same, I think, bar exists for how people should look at using AI tools as they did in their previous work, you know.

Marlene Gebauer (25:45)
Yeah, but I want to pull on that, that thread a little bit, because I mean, this is very much an SME review process and you know, for research, you often have more, you know, junior folks who are not as familiar with the topical area, or you may have allied professionals who also, are not as familiar with the, the, topical areas performing the research and they may not.

You know, they may not really understand whether it’s a hallucination or not. know, somebody that’s got, you know, more advanced knowledge will be able to pick it out quickly. So do you see this changing the workflow in any way in terms of, know, who’s performing this work?

Pablo (26:22)
It’s so weird to get to say this and like be on the side of the luxury, the luxury secondary content. Like so much of my life has been like, you don’t need those luxury secondary content. stop.

Marlene Gebauer (26:26)
I knew this was a Pablo question.

Pablo (26:34)
You’re making up that you love writing. And now here we’re talking with our beautiful experts. And this actually came from our good friends who were taking issue with the legal tech bros. They were saying, they’ll have you believe that AI replaces like Blackstone. First of all, how divorced from the practice of law do you have to be to think that Blackstone is what we’re hearing is like a, how are you right? But anyway, but the point stands that like, mean, I think that having these associates orient themselves with masterly secondary, like second sounds great.

I think that that’s a wonderful place to start and I even put that over Let me just ask Clyde to kind of generally get me up to speed in that area I think you’ll get a better output there, right? There’s different aspects of what goes into law and practicing it, right? There are times where that’s what’s happening other times you’re looking for your fourth case that stands for a certain principle just because string citing it will make it a little easier for the judge and in that case a system like this might be faster, right? Same thing with sort of parallel search kind of thing And I would say this what we’re seeing does

are the walls between the silos. Like you saw, I managed just MCP. said, you guys, now that’s a beautiful moment in our thing, right? Because now this ocean of litigation content, right? Can now be, like can be used in all these other ways too, and sort of things like that. I think that the, if I could push back, I think the Lollobre community might want to see, are there walls around us that might dissolve in the sense that like your, Lollobre don’t think of eDiscovery as their problem, right? They don’t think of transcripts, right?

like they’re guardians of the great common law that’s been handed out and whatever happens out there. And I think increasingly we have to start looking at litigation as this like unified information system that’s happening that is spanning all these different groups. There’s not an infinite number of them, right? There’s only so many. And because it’s that combination that then goes to the judge as advocacy, right? And so I would challenge the folks in the LawLabor community to let the dissolution of walls occur for them, not just in product lines.

might look at, but in how they evaluate what it means to care about legal information.

Marlene Gebauer (28:33)
So, I mean, it sounds like we’re, going a little old school back to the treatises and then also every, which I’m happy about. and, and everybody, know, everybody involved, you know, in, a, you know, working on a matter for, you know, for instance, kind of has to understand all the points and all the workflow of the matter in order to, do their jobs correctly. Fair. Yeah. Okay.

Greg Lambert (28:55)
Yeah.

Well, before we jump on, wanted to, there’s a question that we’ve asked the last couple of guests because, Pablo, you talked about, you know, we’ve heard data’s the new oil or this is the new electricity sort of thing. So we talked with Ken Crutchfield, we talked with Tom Martin, and if that’s true and the…

Marlene Gebauer (29:06)
I know what you’re gonna ask.

Greg Lambert (29:18)
TRs of the world, the Lexuses of the world, the Walters Kluers of the world that have this very specific but very detailed pocket of knowledge that’s very well curated. Do you see the possibility of an open AI or a Microsoft or an Anthropic actually coming in and acquiring some of these companies in order just to get the data? I mean, these are profitable companies.

Pablo (29:43)
That’s

Greg Lambert (29:44)
So and.

Pablo (29:44)
a really interesting question. I had a moment with a judge recently, we were talking about this, and then she alluded to that they were using Anthropic for something. And it was sort of like, oh, Anthropic’s already here, you know? And I’ll tell you, quiet as it’s kept.

I think Dario got access to GPT for before even I did, think, Greg, if we’re really quiet about it. So like, it is this really amazing aspect of what it means to be in legal tech right now that like at the same time that you’re applying and packaging all these things, are simultaneously right. OpenAI is charging $ 1 per government agency. That’s did you see that offer?

I don’t know if we’re authorized, Joe, but I think Sam wants to play hardball. 85 cents, guys. That’s what we’re going to come over. Like, this, the joy of when Silicon Valley starts joining into a party and just like every just it’s so wonderful to watch that aspect of it. And I think that, you know, again, like nobody who is responsibly advising anybody would say, like, don’t go talk to Daria about anything like, I’m sorry, like, like at some point you have to like read that said, yes, I think that the content does matter.

Greg Lambert (30:20)
Hahaha!

Pablo (30:44)
understanding the user flow does matter. We still haven’t come close to building an intuitive system that gets rid of all of the messy parts I don’t like of law. Go find litigators trying to blue book a depot site and tell me that we’ve gotten, we’re nowhere near it yet. I think we have what we need to get to it. I think that as we’re getting to it, we’re gonna see other new flows on it. yeah, so I think it’s amazing that we’re also competing with the Frontier Labs.

that puts on us to be delightful and to be, right, is actually I think the strongest pressure we face out there. And if you’re a lawyer you should be very happy that it’s happening.

Greg Lambert (31:21)
Pablo, you’ve

always been delightful,

Marlene Gebauer (31:24)
Hahaha.

Pablo (31:24)
Well, I mean, it’s

a very fun time. I this is a very good decade to be into the legal technology underpinning litigation, right? I mean, it’s had some developments.

Joel Hron (31:34)
I think it’s a really interesting question, honestly. certainly if you were to look at AI, would say like the more high quality content I can get to put underneath the training, the better. And in many ways, like, I feel like we’ve kind of hit a plateau probably of world knowledge data.

that these models have access to without reaching in and grabbing some of these proprietary things. So to the extent they need to improve the world knowledge of the models, absolutely. What I would say though is kind what I alluded to before. If you look at what has happened to the models over the last year, let’s say, they have plateaued in terms of world knowledge. Where they have not plateaued is in terms of reasoning and logic and code.

And that is where, you know, many of these research labs seem to be focused in terms of their improvement to the underlying model. Certainly world knowledge is necessary in order to be able to like reason and intuit about a prompt or an instruction that was given and how do I interpret that into like a legal context for a plan, for instance, this kind of thing. Like world knowledge is necessary for those kinds of things. But,

You know, I think much more of these research lab focus right now is on this idea of reasoning and tool use and code for good reason. And I think what that means is like for domain specific providers like ourselves, like we need to be conscious about, how do we build the legal tools and the legal know-how and the legal instruction manuals for these agents to operate over with high precision and high trust? And then we also need to be focused on

how do you build the user experiences again to optimize this generation verification loop so that users can build trust very quickly in what they’re doing and what they’re sort of sending off to the judge, right? And I don’t think that the world leading research labs are gonna spend the time building user experiences that do that for lawyers, right? Like that’s a unique experience. And so who’s to say nobody’s got a crystal ball.

I think from pure contents that it’s certainly valuable, but I think there’s a lot of things underneath the hood, particularly in these reasoning models that really domain-specific providers are best to deliver, I think, right now.

Marlene Gebauer (33:48)
actually curious about, you know, I was seeing, was reading the news today about like Iltica and they were talking about, know, Microsoft’s guard where guard guard rail product. And, you know, I’m familiar with, you know, some other guard rail products out there that, you know,

provide safeguards for people who are firms that are using AI. So what are some of the new safeguards or workflows out there that you’re seeing? And how’s that impacting you? What are some of the things lawyers and firms are demanding that be in place in terms of using tools like deep research?

Pablo (34:26)
I admit I’m not as close to that. Joel, I don’t know if you’ve had, like what are the specific policy?

Joel Hron (34:29)
Yeah, I can offer a view.

You know, think it’s a really interesting double-edged sword, actually, Marlene, in that…

When we look at, when we talk to customers and understand what they’re trying to do, they want legal specificity and legal precision in the work. But yet at times they’re like, I want you to help me prepare for this meeting, right? Which is not in scope for any sort of like legal specific action like research or something like this. So we could build a lot of guardrails into our application.

But actually, the diversity of what people want to do with the application is quite wide, which I think is in conflict with building a lot of tight guardrails on systems. so I think it’s really hard to land on this is the guardrail, and we’re going to put it in, we’re going to tighten the constraint, because then you lose out on some of the benefit of the models and what they’re capable of in the first place.

I think that’s a challenging needle to thread, if you will. The one thing I would go back to is like, you know, we talked about deep research, for instance. You know, we were building tools to do verification. Like in a way, this is like a very domain specific guardrail for the model to like check itself. we’re building and constructing tools to allow the model to really check its work multiple times over before it gives an answer. I think that’s a…

a way we build guardrails. I also think about agency on a spectrum. Agency in all cases doesn’t mean I just open up the aperture of the model and let it run to go do everything on its own. If you think about, again, back to DevTools, I’m an engineer, so this is always my thing to go back to, but if you think about how I might do this in cursor, I don’t just…

always vibe out an app and tell, go write this app from scratch. Like sometimes I might say, I want you to help me with this line of code. I want you to help me with this function. I want you to help me with this file, this repo. And as a user, I can scope how much agency the model has in that way. Like I can either put it on a long leash or a short leash. And I think we have done the same thing in CoCounsel.

Like deep research is admittedly like a very long leash problem. This is about like discovery and like, I wanna open the aperture of like what this problem might present and I wanna see all the angles of it. Something like a practical law guided workflow, which we just released at ILTA as well, is not as long of a leash. Like it’s intentionally, I’m executing, you know, unemployment policy. Like I won’t.

I have a known starting point from practical law about how I should do that and what steps I should follow and what information I need. And I want the agent to follow that rubric, right? And so that would be an example of putting agency on a shorter leash so that we get more control over the process. And so Marlene, that’s kind of how I would think about guardrails is somewhat use case specific in terms of how long do we make this leash for the AI model, depending on.

kind how much freedom or appetite for risk maybe we have in the use case.

Pablo (37:32)
Three years ago, at ILTA I don’t know if you guys remember, had the painting of the angels raising the trumpets and then slaying everyone to rejoice. Because again, BERT really, I was really, BERT. And next to me, Damien Riehl pointed out, what about also symbolic AI, right? Sometimes that’s what you need. And I think…

The very lightest of symbolic AI has a place in you know include related federal That’s not something to serve the models gonna do in its own Sometimes you tell it right, you know Like there might be these little moments where it can make a big difference to have somebody who knows what they’re doing Kind of like navigating it I’d like to think that God help us. There’s a few of those because that’s kind of thing will be using jolt To pay for our suffers here, but but yeah, no, so I think that

do think that the, what do we call context prompting now or engineering, right? Like we do need to sort of put this thing together and give it some of the basic structure of what litigation is let it kind of run to the races. But our engineers, mean, the agentic stuff they’re looking at now, these guys are, because RL is such a weird little, like just started as a little nothing and let it kind of work its way up. Like it’s so amazing how little they want to start with almost and then have it kind of develop it on its own. Anyway, it’ll be a balance, I’m sure.

Greg Lambert (38:40)
⁓ Pablo, you kind of talked about this ⁓ at the beginning of the show, but I wanted to, I looked back at our 2021 interview. So this is pre-Chat GPT-3. And there were some things that you mentioned and I kind of want to see where we are now. Back then you talked about ⁓ neural nets, the beginning to vectorize the law was one of the things about training.

Pablo (39:04)
Still going, they’re still going Greg, they’re going

swell.

Marlene Gebauer (39:08)
Yeah.

Greg Lambert (39:08)
And again, transferring

away from the key words to finally the natural language that we were promised in like 2002.

Pablo (39:14)
You guys added rock

and roll music at the end as I’m saying one vector at a time I mean I it’s like I paid you guys to create this like perfect propaganda Like being correct about one thing and one thing in life this guy did get corrected. We put it to music ⁓ yeah, okay. Yes in the black box rights and the right

Joel Hron (39:24)
I’m just

Greg Lambert (39:28)
Yeah. Yeah. So, but

you talked about also, I’m pulling out some of the geekier phrases that you use, like that it’s transforming search beyond keywords into a rich 768 dimensional vector space. I loved when you were describing all of that.

Pablo (39:49)
Yeah, I wish I could say it was my idea. yes,

but I think that I mean, and this is where I could see folks being like, dude, this kid is so drunk on Kool-Aid. Sorry, kid. I like I’m like in high school. But like this guy is so drunk on Kool-Aid that he’s like just saying, let’s just celebrate the fabric that is a 768 dimensional vector. Like just how insane it is that that’s that we’ve created this and then we’re dyeing it with everything ever written and then spinning it off of math.

And then we’re like, their boy just reviewed that privilege, Greg. I’m telling you, we haven’t got public yet. It’s review and privilege. it’s a mean privilege review, Marlene. I think it’s going to start logging with granularity. This is what we’re dealing with just with today. I’m not hoping for the, you know, this is what right now we’re going to be able to do. And it’s, you know, it’s early days. It’s a lot easier to do prototypes than scale-proof concepts, but.

Marlene Gebauer (40:24)
you

Joel Hron (40:31)
We to a lot more

Greg Lambert (40:34)
Yeah.

Joel Hron (40:36)
dimensions in 768 pretty quick too.

Pablo (40:39)
Joel, you’re not supposed to tell them that. now they know. Yeah, OK, so yeah, black box. Let’s start with black box. It’s worse because there’s it’s worse. Here, let me argue why it’s worse. OK, first, now that it’s like an issue of national security.

Greg Lambert (40:39)
Yeah. It sounded great at the time. now to go ahead.

Yeah, yeah, let’s talk about that.

Marlene Gebauer (40:50)
Black Box is

Pablo (40:58)
what happens right now. It’s like we have to win the global war of it, right? People are less transparent about progress necessarily, unless willing to like kind of put their stuff out, right? The biggest breakthrough that we’re celebrating, the reasoning models.

Joel, correct me if I’m wrong, but the main thing we know about this is because China, a team in China, put a paper out for their version of it. And then you can like dig through the tweets to find the one OpenAI guy who’s like, all right, they figured out the same things we did. Like that’s the level of like Kremlinology. Like just trying to like, maybe I can piece together, right? So before it was like, we don’t know how they work because it’s just not something human mind can comprehend. But like if people knew they’d tell you. And now I feel like it’s like they don’t know how it

and they’re not going to tell you to the extent they do. Moreover, and this is where it gets very dangerous, this chain of thought that comes out, there’s fierce debate about whether that’s actually showing you what the reasoning is. How much can you rely on it? And Anthropic as leaders in doing these cool experiments where they tell it like, hey, I hacked into a computer and that’s why I know this x.

and then it solves the problem and in the chain of thought it just leaves out the part about X even though it’s not able to solve it without X. Right? You see what mean? So they teased out and now there’s this paper that came out, signed, like endorsed by Ilya Sutskever, like just the most perfect group of people you could ever seem to get together for a paper saying to everyone you need to monitor the chain of thought in certain ways.

But if you train it to like punish it for bad chain of thought, it’s just gonna learn how to do it without telling you about it, right?

Greg Lambert (42:32)
It’s going

to learn how to sneak out the second floor window.

Marlene Gebauer (42:34)
Like teenagers. Exactly.

Pablo (42:36)
Yeah, so like

words, black box, yeah, yeah, go ahead.

Joel Hron (42:37)
There’s this idea of goal hacking as well, which is kind of similar to what Pablo’s referred to, which is like, like on a coding test, for instance, if you tell it, like, write this code to pass this test, here’s this repo. It’ll go like, potentially write some code, and if it can’t figure out how to pass the test, sometimes it’ll be like, well, maybe I should just rewrite the test.

Now I pass, okay I’m done. So there’s this idea of like, what is like the right constraint that the model should innately know? It can’t break this constraint in order to achieve this goal. And it’s really interesting because like, I mean if you follow that like to the you may get into like ethical conversations. Like this model needs to have its own ethics in order to know like, okay, if I’m goal oriented, like what ethics do I need to have? What can I not?

Pablo (43:01)
Yeah.

Joel Hron (43:27)
do in order to achieve this end-of-state goal. So I think that that’s kind of like one of the maybe like leading areas of research. And again, like transparency work, certainly Anthropic, I think a leader in that area.

Greg Lambert (43:39)
Can

you even build ethics into it? Because every time they try to adjust something, and I think Elon Musk got in trouble with this, it was like, well, we’re going to let it do whatever it wants to do. And next thing you know, was the or an Anthropic, I think, story somewhere. What you were telling, Joel, was like, they were telling it.

Pablo (43:44)
meet.

Greg Lambert (44:03)
Make sure you’re not doing this, but then it found ways to kind of like do it without making it look like it was doing it. So. ⁓

Joel Hron (44:08)
find a loophole.

Pablo (44:08)
Yeah. it’s a rowdy little CSV

file. You tell it, I don’t want you to decide the case. I just want you to report distortions. And then every once in it’s like, hell no. No, no, no, no. I got this one. And you’re just like, take like, what? And it’s like, sometimes it happens and sometimes it doesn’t. So it’s a very weird situation we find ourselves relative to the transparency, right? And as I said back then,

you know, is not, it should be troubling in some ways, especially given the increasing role they might have, right? But that said, we faced this before. Remember the shaman to go back to the metaphor, right? He would rub the berries and now the rash went away and I didn’t know how it worked, but man, look at that arm. like, so like humanity has survived through times where we didn’t fully understand mechanisms, but we did understand.

vet output. Now depending on the use case it might be that that’s impossible, right? Like certain things you might not be able to let go through a black box.

Joel Hron (45:02)
Bye.

I’ll say these are all questions for the research community for sure that are like top of mind for many in the field. You know, think from an application building standpoint, it goes back to kind what we talked about before. think building experiences that elucidate everything you can about what the model is doing and why and allow users, you know, clear visual signals and cues as to like what they need to look at and when and how to go deeper when they need to.

I think this is kind how you work around some of those issues until they get solved in the broader research community. But yeah, these are quite maybe like academic edge cases right now.

Marlene Gebauer (45:44)
All right, well, we have come to the point in the podcast where we asked the crystal ball question, and I’m very curious to hear what the answers are from each of you, but what change or challenge is the industry gonna face in the future that we need to start preparing for now?

Pablo (46:03)
She’ll go first. And are you actually also speaking on behalf of TR? Because they always, I have this long thing I have to read that says it’s my views and not that, no, I’m kidding. It’s actually two pages at this point and it requires the listeners to consent. You actually have to consent to it and say, yes, you understand on the screen. But no, but Joel please go first. I have some thoughts where I think like what’s after agentic but let’s, I’d be curious to hear your thoughts first.

Marlene Gebauer (46:10)
disclaimer disclaimer disclaimer

Joel Hron (46:10)
Hahaha!

Greg Lambert (46:13)
I’m sorry.

Ha

Joel Hron (46:26)
Yeah,

they usually don’t give me the choice whether to speak on behalf of myself or the company. ⁓

Greg Lambert (46:33)
Yeah.

Marlene Gebauer (46:34)
We will acknowledge that you are both speaking on your own behalf and that this does not reflect the views of Thomson Reuters.

Greg Lambert (46:37)
out.

Pablo (46:40)
If you like

it, if you like it, go ahead and let it in directly, mostly, yeah, all right, all right, go ahead.

Greg Lambert (46:44)
Yeah.

Joel Hron (46:45)
I’ll give actually a non-technical answer if I can. think agents has a long run way to go. mean, honestly, a year and a half ago, a lot of people were talking about long context. I was talking about agents. I thought a while ago this was going to have a profound impact and I think it still has yet to even play out as to sort of how deeply that will impact software in general, not just legal, but software industry full stop.

I think for the legal industry, and would say the same for my own industry being technology, think talent and workforce is one of the big things that, you know, is kind of ripe to change in this environment. Like I said before, I think there’s a lot of work that agents and AI will be capable of doing, but the hardest parts of the work will become more common and more important.

because those will become, I think, more of the day-to-day of a user. It’s making these hard decisions, hard directions, hard re-steers of the AI model. And I think it will surface the need for really great talent in those domains, and that those people will drive your organization 100x relative. And so…

I think from an organizational design standpoint, a talent management, a talent development standpoint, I think that’s a different operating model for how you think about building teams and things like that. And it’s probably one of the biggest changes that think folks have to kind of start thinking about a little bit.

Greg Lambert (48:17)
All right, Pablo.

Pablo (48:17)
For me, think the next buzzword should be ambient. And what I mean by ambient is an AI that is, maybe analogy would be your metaphor would be your spell checker, right? Your spell checker sits there watching every word you type and yet you don’t mind it. And it just shows up only if and when it needs to just help you out a little bit because something got screwed up. It’s very easy to override, right? You have a nice relationship with this little intelligence that understands that you’re going to be misspelling Fahrenheit, right? Maybe even three times before you finally like, right?

Marlene Gebauer (48:18)
We’re ready.

Pablo (48:46)
So what does it mean? then and and you know, I’ve last decade, the whole thing I was only do is like you could take a little brief and load it up and see if there were missing. Right. I think what we want to expect is what are those things that there’s just no way you don’t want to have happen. Right. If somebody distorts and misquotes a case or fabricates. Right. Like it’s not right. So what is the system right now? Well, download it and drag it over. And now, you know, we’re getting the NPCs kind of. Right. That kind of stuff, I think, should just be like appearing.

Greg Lambert (48:46)
or four times. ⁓

Pablo (49:15)
Basically, right when there’s that kind of thing and we’ll have to calibrate it different people will want it set to different volumes, right? I think that that’s gonna be one like really Important way that this actually starts to spread a lot. It flips the risk calculus completely because Humans are missing privilege humans are humans are filing things where they say there’s an exhibit D and there’s all these different ways they’re screwing up from substantive to Administerial and if you can have something that magically prevents that why not right?

What’s like it’s not right. I think what we’ll be having the deepest controversies about and maybe we should is appellate judicial drafting, creating binding precedent.

And I have this romanticized view of writing. think wrestling with the blank page is like Jacob wrestling with the angel. I’ve got all these weird things that have no tie to ROI or anything. I really worry about a world where the first draft is actually just completed by the AI and then the judge is sort of chiseling that into what they want. I think we need to be very careful with it.

I used to feel that way about trial court judges until I talked to them. And now I’m like, get those people to help. Like, oh my God. And I sort of made my peace with that. I mean, it’s not as binding. It’s not precedential So maybe there’s a different calculus. But I think that’s where we’re going to need to be really careful is how we fold this into the judiciary. know great folks like Shlomo Clapper are looking at it. Ross Guberman, there have been folks who have been contributing to the informatics of the judiciary.

Greg Lambert (50:29)
Yeah.

Pablo (50:51)
Thomson Reuters can and should continue to play a role and to help ⁓ enhance their. So yeah, those would be my two things. Ambient, and then we’re going to be all talking and there’s going to be conferences about what does it mean. And I would invite the law librarians there, especially who I think have a unique perspective in terms of intangible things that can be lost. This diatribe sort polemic thing talked about like, they’re going to turn the common law into a data set, right?

And I think that in a way, I mean, I think they touch on something there, right?

You know, molecular biologist turns humans into a molecular pathway for a purpose, right? I don’t think you’ve stopped molecular biology because if you only have that perspective, would, right? In the same way, I think that like we are going to need to think about the human aspects of law while we take the data set side and just optimize it in ways that I do think we’re going to get to the good stuff. I’ve been very excited for a while.

Three years ago, had Cravath talking about using language models in your discovery and Fisher Phillips Samantha Seedon was talking about real time impeachment, right? They were just type, right? You know, are we three years past that yet? No, I don’t think we are. I feel like we’re still, you know, but you see bit by bit, like walls are coming down, people are sharing data, companies are reorienting themselves as more tech focused. All the things that you need to start happening are happening. And…

I think this is going to be a fantastic next five years for us, for those trying to make law better.

Greg Lambert (52:19)
Well, let me test

out one other question that Marlene and I, well, it’s Marlene’s idea that we want to test. Joel, let me start with you. What is a place, or where’s a place that you go to kind of keep up on what’s happening in the industry for you? Is there a good place that kind of helps you keep up with things?

Joel Hron (52:41)
Yeah, it’s a good question. I mean it’s hard to look past X these days. I’m not even that active on X. But I see a lot of interesting stuff get posted on there and it’s good breadcrumbs. I think technical reading though that I like. I see a lot of good just technical blogs get put up on Medium.

from a lot of different companies on. Yeah. So I like that a lot. I’m certainly active on LinkedIn too. And I think that can be a good pathway for some things. I think, you know, I see a lot of good stuff come through medium blogs and there’s a lot of good authors on medium who write pretty regularly about these topics. And I think it can be quite good. You know, I…

Marlene Gebauer (53:04)
I’ve seen good stuff on Medium too. ⁓

Joel Hron (53:29)
I would also say that like, I don’t know, many people aren’t probably reading academic papers, but like, I try to read at least a paper. I try to read a paper a week. I think, I think it keeps you up to date on like what’s happened in the research community. And it’s like one of the best signals of like, kind of how people are thinking and where things might be going. So, ⁓ I certainly try to stay up to date on that kind of stuff when I can too.

Greg Lambert (53:36)
Pablo does.

Pablo (53:37)
Hmm. That did.

Marlene Gebauer (53:37)
was gonna say, yep.

Greg Lambert (53:53)
Bob Lowe, what about you? ⁓ What’s place you go?

Pablo (53:54)
I think

Francois Chollet, C-H-O-L-L-E-T, used to be at Google and now is doing his own effort. I think it’s a very balanced kind of measure, right? Not hypey about it. Some said he was a detractor, but I think if you read him closely, he’s not. He’s just describing it well. He’s certainly good. Jason Way at OpenAI recently wrote a great piece about mapping out reinforcement learning. Unfortunately, Joel’s right, X.

is a source. And so it’s this horrible thing of the mix of like the dumbest thing you’ve ever heard. And then all of sudden, like somebody being like, no, no, no, you’re like, that’s a genius thing you just read. But I, you know, I’d be the first to say that like, you know, the the folks behind these models, actual scientists building it, it’s very important to talk to them. And a lot of them, mean, to listen to them, a lot of them do put out stuff. Jeff Hinton. Oh, have you guys hung out with Richard Sutton at all virtually the Turing Prize winner for reinforcement learning?

He is so good at sort of this combination of the majesty of everything that’s happening, that there is intelligence in this, right? With the like, take a look around guys, do not take yourself seriously. Like that is not what the universe is kind of telling us from data that like, humans are not forever. You’ve never had control of anything. just like, like just stop like, and I really, he’s clearly a guy who’s enjoying things. And I would recommend Richard Sutton, not just his stuff on reinforcement learning, but occasionally he’ll wax poetic.

He’s quite good and obviously a very brilliant, brilliant guy.

Greg Lambert (55:13)
Nice.

All right,

well, Pablo Arredondo and Joel Hron thank you both for taking the time to talk with us on the Geek in Review.

Pablo (55:23)
Alright,

see you at GPT-6 or if there’s some other big conceptual breakthrough, let’s do it there and before it goes out on six. But thank you guys. I would hope so. would hope so. Alright.

Joel Hron (55:24)
Thanks Greg.

Greg Lambert (55:29)
Sounds good. And we’ll make fun of all the stuff you said today. ⁓

Marlene Gebauer (55:32)
The last time. ⁓ and of course,

thanks to all of you, our listeners for taking the time to listen to the geek and review podcast. If you enjoyed the show, share it with a colleague. We’d love to hear from you. So reach out to us on LinkedIn and BlueSky we may need to get back on X.

Greg Lambert (55:47)
Sounds like we may need to get back on X.

Pablo (55:51)
I did.

Greg Lambert (55:51)
Pablo and Joel, for listeners who want to learn more about what you guys are doing at TR and with CoCounsel and other projects, Joel, where’s a good place to reach out?

Joel Hron (56:01)
Yeah, most of the stuff that we’re doing, I try to keep reasonably active on LinkedIn and post updates there. You can also check out a lot of stuff on our website. But most of the major stuff that we do, I try to keep active on LinkedIn and include links to get to the good stuff.

Greg Lambert (56:17)
Pablo?

Pablo (56:18)
Emily Colbert and Matthew Hergerty are the ones with their fingers on the pulse of what’s actually happening at CoCounsel ⁓ So I would definitely see if you could find them And yeah, think it’s just something you gotta, things are moving fast. So, you know, keep at it.

Marlene Gebauer (56:31)
All right, and as always, the music you hear is from Jerry David DeCicca Thank you very much, Jerry. Thanks everybody. Bye.

Greg Lambert (56:36)
All right, thanks.

 

LexBlog