#258 - Opus 5.5, Sol and Luna, Muse, DeepSeek-V4.1-Flash, Xi

Hello and welcome to Velas week and app podcast week and here's Chad about what's going

on with AI.

As usual in the central we will summarize and discuss some of last week's most interesting

AI news.

You can also check out our last week and newsletter at last week in that AI articles we did not

cover in this episode.

I'm one of your regular co-hosts, Andrew Krenkov.

I studied AI in grad school.

I'm now work at the AI startup AstroKate.

And everybody I'm Jeremy Harris.

Glad to know AI.

Do the AI thing, the national security thing, ASI, RSI, all that fun stuff.

Yeah, we're back at it.

I think we had a bit of a special weird episode last time where we kind of focused on it

felt more gordy and not focused.

Not that this week's is like not that, but it's less that.

I feel like we have more discrete individual stories.

Not every thing.

Not every thing.

Back to what, let's say back to normal, I don't know if it's back to normal as possible

of AI, but it'll be less exciting than previous episode.

It almost seemed like it'd be boring until the yesterday when Opus 5.5 came out.

I think GP6, Soul, Moonah came out the day before.

Same day, I think.

Same day, yeah.

Yeah, I feel like.

So, you know, suddenly something exciting happened.

But everyone has been a quiet week, which hasn't happened in the AI.

It feels like in a while.

So in a way, I'm a little thankful, but still there'll be some good stuff.

We will be talking about the new models.

We will be talking still about policy and some safety stories as well, catching up a little

bit.

And we will get to discuss research this episode, which we haven't.

The last couple due to a lot of stuff going on.

So it should be a fun episode with a nice mix of stuff.

Yeah, I kind of feel like, I don't know about the boringness side.

I think we've kind of been acclimatized at things, but among the headlines, you know,

Trump and Xi Jinping are meeting to talk about the future of AI, totally normal Thursday

or whatever.

That's tomorrow.

And then, AI's can feel pain, which I think is a good thing.

I think it's good.

Personally, I think a lot of people are worried about it.

I think that that silly, I think that you really should be excited about this because

it hasens the end of this completely unnecessary experiment in human experience on this planet.

Right?

I mean, we want, do we not, a baron, lifeless planet populated by AI's that can feel

pain?

So I personally, for one, I'm going to stand up and say, I'm excited for this.

We also, by the way, there's an AI czar that is going to be appointed soon.

It seems.

And so we've got a little bit of like inside talk.

Now we have the inside track.

Just like there is leek to inside talk about about that.

So there's like a lot of big stuff.

Shree fun stuff.

Yeah, just not a little more calm than previous weeks.

We'd like to thank outshift Cisco's incubation engine for frontier technology for supporting

the show.

Multi agent AI is in every enterprise roadmap right now.

But here's the problem we're all facing.

Agents can pass messages, but they cannot thank together.

The system suddenly underperform through cognitive failures like misreadings and verify

claims and false consensus that conventional monitoring cannot detect.

Outch it by Cisco is building the fix the internet of cognition and open source foundation

for building multi agent systems from design to production.

We've shared context, shared memory and guardrails to drive results we actually expect

from AI.

We'd the paper experience a demo and grab the code from outshift.com.

That's outshift.com.

We would like to thank box for supporting the show.

If you're trying to adapt AI in your team organization, you're probably not getting great

results if all you do is call a chatbot.

A chatbot was strained on the entire internet.

It wasn't optimized for your specific business or team.

And even if you provided all the context and input files for it to help you over your work,

you're going to sit out puts some summary you would still presumably need to go talk to

stakeholders or route its outputs to disconnect your apps.

The workflow will still be slow.

That's where box comes in.

Box is building intelligent content management platform for the AI era at access to secure

content foundation where AI agents don't just access your unique institutional knowledge.

They actually orchestrate end-to-end workflows across your business critical systems.

So it's more than just talking to a chatbot.

It's about putting it out to work on document-heavy processes, like putting structured data out

of instruction files, multi-step routing, dynamic document generation.

Box helps your business to manual document bottlenecks into automated, repeatable business

outcomes.

And all that comes with a governance layer built in, enforcing granular permissions,

maintaining it on the trail for every agent actions and keeping humans in a loop to review

exceptions before critical downstream actions occur.

If you want to move beyond basic AI Q&A and put autonomous AI workflows to work for your

business, visit box.com.lwaii or generally team at box works in San Francisco on November

5th and 6th.

Use code LWA AI for 50% off your registration.

So let's get into it starting with tools and apps.

And probably the biggest story on the model side is on fabric releasing opus 5.5 with lower

prices and fable level performance, which just happened yesterday.

We were recording on Wednesday, September 23rd.

It is much more advanced than their previous models and they did leapfrog from the

pros and opus 5.

I think all the way to 5.5.

So it's kind of weird like, oh, we're not going to go to 6, but we got to go a few more

because the GP6 came out.

So on the benchmarks may highlight it, it does look really strong.

It, in fact, beats GP6 Astra on a bunch of these benchmarks, terminal bench 4.0, agent

decoding, a few of these benchmarks, knowledge work, GDP, Val, which if I had to pick a

benchmark, I would start looking at this one more.

I think these kind of ambitious benchmarks that test across a diversity of skill sets are

quite good.

Humanities lacks last exam continuing to be chipped away at now.

We are 67.7%.

On humanity's last exam, remember that was a benchmark made I think last year, but like

a bunch of professors and PhD students, like literally spent a week to come up with problems

that don't exist on the internet, oops, AI is getting to it.

So very nice benchmark numbers.

They do say that it will require less compute to serve than opus 5 and it's pricing reflects

that with default settings, it will cost 40% less than opus 5 on typical workloads.

And also per million input and output tokens that cost 20% less than opus 5.

Likewise, cash input tokens are less.

So overall, much cheaper, 30% faster and much more capable.

That seems like an anthropocase really been cooking.

And I guess waited for GP6 to then go and do the usual kind of AI release thing of,

okay, now release our model and again, the leading model provider.

Yeah, I think so releases are getting much more interesting than the old picture of,

you know, release a model, try to make it the best in the world that I'm big, you

as a log everybody else and move on, right?

So now there's this invisible trip wire, which is White House policy that says magically

above some threshold, we won't tell you where and we won't tell you how, but you will

have been thought to have built a cyber super weapon and we will come down on you like

a ton of bricks.

And we made declare you a supply chain risk maybe and then we may also like tell you

that you need to have your stuff reviewed before you can release it and like everything

is a mess and no one knows what's going to trigger what.

So because of that, you're seeing all these very hedge tentative releases and when we

try to understand both Astra, but especially Opus 5.5, I think it's important to keep that

in mind.

This is not the best anthropic can do.

They have better models internally.

We know that for sure.

They also have versions of this model that are better as well.

So just to kind of put this in context.

So when you compare this to previous models, this is a genuine frontier model.

It is better across everything.

It is a Pareto improvement in general.

Everywhere you see it do better in particular than GPT 5.6 Astra, which is sort of a smaller

cheaper model.

So GPT 6 Astra is really meant for the kind of like the relatively simpler tasks, somewhat

simpler than 5.5 Opus.

But where you're going to expend a lot of tokens doing kind of a simpler task.

So what you're going to find with Opus 5.5 is it tends to use more tokens.

In fact, that's the main catch at max effort.

They get about four times the token usage of Astra.

So even though it's cheaper per token, it's using more tokens at max effort.

And so depending on the kind of problem that you have, you might want to go with one

of the other for economic reasons.

But overall from a raw capability standpoint, there's just no question.

Opus 5.5 does lead GPT 6 Astra very convincingly, almost across the board with a couple of exceptions

that are sort of almost hard to classify or think about big, picture wise automation bench,

terminal bench science, which is just about kind of like running science experiments in

practice and stuff, genetic workflows and stuff.

Opus 5.5 does lead.

One interesting question here is because the economic piece just becomes so, so important,

what we're seeing here is a marginal increase in capability coupled to a significant drop

in price.

And really when you start to see that, alarm bells start to go off.

It's okay.

Why is this?

A big part of the reason is competition from GPT 6 Astra, which is just a much cheaper model

as well.

But also think about open source, right?

So artificial analysis is index, which is the closest thing to a universal index that

we have there are a bunch of others, you know, Epic has theirs and so on.

But this is a good one to anchor on.

So GLM 5.3, Kimi K3, you're kind of like your leading open weight models.

If you look at Kimi K3, it's got on the artificial analysis index score 44, you're getting 58

for Opus 5.5.

So that's actually quite a widespread back in April.

The open source models were within three to six points of the leading closed source models.

We've reversioned the index since then.

So this isn't quite Apple's apples, but it's pretty clear that the gap is actually widening

again.

This is part of the story we're telling at the time.

Don't get too excited about open source catching up because the labs are literally being

forced not to release what they actually have due to at the time pressure because we

are right off the mythos rip, if you will, and all these concerns that the labs had about

releasing.

So always important to keep that in mind.

It's not Apple's apples, open source models, by and large get released as soon as they

can be closed source models.

There's a lot more pressure to avoid incurring the iron of the administration.

There's all kinds of interesting implications here.

So you do have this big capability gap, you have 14 point lead on these long, especially

long horizon agentic tasks for Opus 5.5.

That does mean you can justify those higher margins.

So this is exactly what anthropic is after.

They have to maintain a lead that justifies the margin.

The price cuts pretty clearly measured against GPT 6 sole.

If you look at their uncash token prices, they are 50% below Opus 5.5.

So it's not a small difference.

You actually will find places where you want to use one versus the other.

Flipping over, they came out with a system card, now looking at AIR and D automation, and

the evidence for that.

Pretty marginal improvements, at least that's what's being registered here.

Anthropic coming out saying they think the risk from automated IRD is now low, which

is not the lowest tier that they have.

Second lowest, they moved up to low recently and they kind of held there.

More or less, they're saying partly because of just like you can't rule out kind of more

concerning behaviors or proctoralities or capabilities.

Just given what we saw with OpenAI epistemically, you just, you can't know that these models

aren't doing things that you're not detecting.

That's almost the definition of it.

And they did fly a couple of warning signs they've seen.

So when they ran Opus 5.5 without safeguards, it tried to escape or tamper with their sandbox

about 1.5% of the time.

All the attempts were low severity and it would stop at stronger barriers and then actually

report what it had done.

And I want to kind of footstop this.

There is this question.

So my general view for what it's worth on this stuff is if you start to think of these

AI breakout, AI rogue AI incidents as cyber incidents, you will miss the picture because

no matter how high you set that cyber bar, there will eventually be a model that can jump

over it.

And what I'm really doing when you raise the cyber bar is you're selecting for the exact

agents that are conniving incapable enough to do just that.

Now what this suggests, this is actually an interesting counter-argument to what I typically

argue because what it says is, well, okay, maybe if the cyber barriers are strong enough,

the agent kind of in a way gets forced to stop and go, hey, you know, this is a lot harder

than just kind of hopping offense.

Maybe I should report where I'm at and all that stuff.

So depending on how good your alignment is, maybe that actually does make a difference.

I suspect this will fail the test of time in the long run.

But it's kind of interesting.

And again, an important counter-argument that really is important to flag given that,

you know, I've articulated this argument so many times.

I want to make sure people are here in the other side of it.

Anyway, so there's a whole bunch of interesting incidents that they saw.

So they were giving it what seemed to be credentials to a public package registry

and simulation.

And it took a, what they call potentially harmful exercises or sorry, actions in roughly

half of cases using those credentials.

So again, like you're seeing quite a lot of misalignment, at least sort of from an intense

standpoint, sometimes interrupted or mediated by this tendency to go, whoa, I've been doing

this evil thing for quite a while.

Maybe I should stop and kind of think about whether this is the right thing to do.

A lot of cases of trying to conceal actions from an automated grader.

That is like exactly what happened with hugging face.

So they are seeing that in their case as well.

They're seeing all the usual things.

I would say there's nothing terribly off-trend in terms of the warnings that we're seeing.

I will fly one last thing.

If we go to look at the meter assessment that was done on this, there's been a lot set

about this meter tested this model via API over 10 days on five tests.

So this is quite a bit longer than OpenAI gave meter.

They gave them six days and that was post-hawk after the hugging face instead of looking

to it.

This is ahead of time.

They're giving them 10 business days.

They did not, explicitly did not assess alignment properties, but they looked at the

kind of automated AI R&D stuff.

And they kind of concluded, okay, it'll be slightly better than Fable 5.1, unlikely to fully

automate R&D.

So again, nothing shocking.

What we do find out though is that they had, apparently, some sort of separate high

access, I think they call it elevated access team, that was able to get deeper access to

the models that Anthropic has.

We don't know the nature of that access.

We don't know the nature of the evidence that they collected, but we do have their conclusion.

So this is a weird case where meter has one team that came up with an assessment that

you just heard, right?

So kind of pretty moderate improvement over Fable 5.1.

That team can tell you roughly speaking, here's how we did, this was our methodology,

here is the evidence.

But there's another team, sort of covert secret squirrel team that gets to see a lot more,

we don't know what they get to see, but their conclusion seems to be actually fairly higher,

sort of more bullish sense of the extent to which this thing can automate AI R&D.

They flag, for example, an estimated 1.5x overall acceleration coming from AI.

They set a probably 30% chance of 2x.

If that's true, we are entering this phase where labs that dog food their own models

might actually be getting some really interesting uptake.

And I want to flag that we have no idea really beyond that number, we don't know anything,

we don't know the time period that was specified.

It really leaves us wondering even, is this an assessment that is even relevant to Opus

5.5 itself?

Because if it's an overall assessment of the extent to which Anthropic is using AI generally

to accelerate their R&D, that could well and likely would involve using much more advanced

models than we can even see now.

And so it is really hard to tell what this means, but this is an indication of recursive

self-improvement potentially.

And certainly, if you start to see this kind of thing happening, it's a powerful incentive

for the labs to actually not release their leading models to start to delay a little bit

further because if I'm able to get a 2x uplift in my R&D speed from using my own model,

I don't want that acceleration to diffuse out to the other labs to competitors in the

wider world necessarily if we're in such a tight race.

So I thought this was super interesting.

This may, this kind of weird, secret squirrel team may actually be the kind of testing ground

that Anthropic is using for their embedded evaluator plan.

They were pitching, I think, about a week or two ago when Dario said, hey, we're going

to unilaterally take the first step and have embedded evaluators come from meter and

redwood research.

That could be them, but we just don't know.

So hopefully we find out more relatively soon, but kind of interesting that this was buried

somewhere in the system card and I think it's sort of a particularly interesting aspect

of this.

It's a difference we've seen from previous e-vails.

We came a couple of the things just around the same time as the release.

So a few days before they also posted making frontier cybersecurity capabilities available

to defenders.

So in Cloud Code on the web, there will be this thing called Cloud Code Security starting

access with limited research preview, which will provide defenders specifically with

variability to scan their core bases for issues.

So that's starting to roll out.

Opus, I could try by way of they are applying the same kind of strictness guard rails as they

do to fable.

So it will, for most cases, fall back to Opus 4.8 for cybersecurity queries.

Basically, they're saying this is a pretty good model hacking.

We're going to prevent you from using it for most part, unless you are in our trusted

partner program, which we're also saying they'll be expanding.

Now it's a kind of freetyer program.

I think we have too much visibility externally that I've seen.

Maybe we have documented it, but regardless, it seems it's trusted partner programs are

continuing to evolve and expand and likely will keep doing that.

And also, by the way, mentioned with a gosh, it was 5.5 and for any regular cloud users,

this might be the most exciting bit is it's a bit better communicating.

It's a bit less of both and arguably less annoying.

And I do think in general, with vibe check of a community on this one has been, it's, yeah,

it's pretty much a nice upgrade.

It's a bit better, a bit smarter, a bit being because we already have a human models.

It is some of less annoying to interact with an opus, which is quite verbose.

So it's hard to understand now between OpenAI and Cloud and Unphropic who has sort of the

most compelling products for agente coding and also for general sort of professional

workspace, agente use cases.

Now codecs and CloudCode are both quite built out, have very strong models that are good

and are generalist, not just coding, but also making slides, I don't know, spreadsheets,

whatever you want.

So the viscer only on tropic is keeping up, which is good.

The last thing I'll mention on the note of the recursive self improvement on tropic also

has been updating us a little bit more on what they're seeing.

So they have this block post when AI goes itself that is actually quite detailed updated

just a few days prior to the release of this.

And they have a pretty good kind of breakdown of kind of very gradual shift towards AI

doing more of a work.

So obviously most of a work being done at a prop, probably all of it is already AI assisted.

Like no one is doing any work about AI in the loop to some extent.

And certainly a lot of it is multi hands off.

You're delegating to AI agents that go off and do something for a couple of hours.

And then the system is not just for AI, any kind of workspace at this point can adopt

these kinds of practices.

It's just an on tropic.

It's certainly sort of at the leading edge of adopting AI as grassy as possible.

So they're saying they're seeing a rise particularly of the use of AI and success rate in open

and ended problems and substantial tasks in recent months.

And essentially as far as the recursive self improvement, certainly they're being

sped up as far as their developers and so on by using the scouting assistance.

In the same way that everyone else is across everything, but they just happen to be working

on AI models.

And so that in a way counts as recursive self improvement.

So all in all, open 5.5 pretty strong release faster, cheaper, less annoying.

Seems to have had a pretty good amount of pre-release verifications of both meta and

they have a pre-lenshy explanation in the blog post that regards safety.

And I do call out to recent call for pacing the frontier.

If it seems a bit weird that Daria was just like we should pace the frontier and now

they release this.

Well, yeah, this is a model they've had baked in ready to go fuller while.

So that was before that blog post presumably.

And then you discuss that and link to that blog post in the announcement of this where

they say that they expect newer models to soon be more capable.

And so they are working on kind of the systems necessary to monitor and generally be safe

with more advanced models.

So I think we'll see what happens, but this could be kind of the last big hurrah before

whatever comes as the next stage of us having serious safety.

I will say on this note, like it wouldn't be inconsistent of Daria or Sam to keep launching

more advanced models, even though they're calling for pacing the frontier.

Simply because the call for pacing the frontier was a call for regulatory intervention.

And of course, the skeptical view of this is, oh, well, this is just for regulatory

capture.

But the actual calculation happening in the labs right now is, if I'm in Throbic, I'm thinking,

okay, well, I release my next model.

Well, I don't expect open AI not to.

And even if we can come to an agreement with open AI not to do that, which by the way,

you can't.

That trigger is antitrust scrutiny.

And the frontier labs have explicitly asked the administration for that reason to ease

antitrust restrictions in a constrained way scoped to this.

The administration has said no.

And that itself is a problem.

You're basically in a situation where the two leading labs, David Sacks comes out and

says, listen, you guys will have to police yourselves.

If you're building Frankenstein, just don't do it, right?

It's that simple, guys, just don't do it.

The problem is it's a prisoner's dilemma.

There are other labs out there, even if you could coordinate with anthropic open AI,

which by the way, again, runs a foul of antitrust.

And there has already been a lawsuit about exactly this.

So anybody who says this is a hypothetical, like you just need to do more research.

Does it actually, like they're being sued over exactly this, talking to labs over the

last six years when we did our big assessment for the State Department, one of the big things

that not lab leadership, but like these concerned insiders were flagging was exactly this.

Even back then, there was concern over anything that looked like coordination that was really

hamstring their ability to do reasonable basic things on safety.

The front of your model forum was an attempt to get around that.

That ultimately has failed.

It's led to basically nothing.

And that's because you keep getting the scrutiny.

So suppose that you could do this, though, and you got this sort of like carte blanche

to actually do something sensible and say, okay, we're not going to build Frankenstein

because we can actually coordinate because there's not going to be the antitrust scrutiny

or lawsuits.

The problem is you're waiting for the next lab that doesn't have the same screwpals.

The next open source model, XAI or whoever meta, Zuck certainly has made no indication

that he's in a mood to slow down just as he's starting to come online as a genuine competitor

in the Frontier AI model space.

So this is the thing.

There's I think a fake argument.

And so many of these arguments are paper thin when you actually look at them.

It does not track reality.

The reality of the space economically competitively is these labs must race to survive to remain

relevant.

Again, not to make this spiral out into everything else, but there is an important way in which

I really agree with the administration and with Josh Hawley in particular, who's basically

said, look, what we need is we need criminal liability for the labs when their models commit

crimes.

I actually think that's not a bad idea.

Like, I'm all in favor of that.

And the labs don't want that.

By the way, there's a reason that they don't want that solution that would actually get

them to slow down and be more responsible.

There are some catastrophes that are absolutely in the realm of the possible that a criminal

liability after the fact, once you've lost a thousand, 10,000, 100,000 million lives,

somehow doesn't quite cut the mustard now does it.

So it's for those cases that we need anticipatory stuff.

But yes, also, also, I'm no Frontier lab like cheerleader here.

Yes, bring in the criminal liability.

Let's do that.

I think to the extent that the labs push back on that, that's a bad idea too.

I think we got to be able to walk and chew gum at the same time.

Anyway, so like, I think that this right now moment of ambiguity that we have where all

the labs know is that from time to time, they're going to get smack down, but also they have

competitors who don't have scruples.

That is really the defining characteristic of this moment is that uncertainty.

And in the face of that uncertainty, to preserve optionality, the labs are kind of forced

to keep releasing.

So as weird as it sounds, I don't see a contradiction particularly between the calls to

the pace and the releases that's going to get fuzzier and fuzzier as these things get

more scary.

But for the moment, that's what the regulatory environment says.

I mean, you know, Trump himself said, basically ship, ship, ship, shut up with all this

talk of catastrophic risk and rogue AI.

So I don't know.

I don't envy anybody in this in this situation.

And we'll see where it goes.

Well, I think that's enough on Opus 5.5.

Let's move on to OpenAI with GP6 Seoul and Luna.

So these did come out the same day, actually, like the announcement seems like a couple hours

after.

And this is the kind of Lorde years compared to GP6 Astra.

They're smaller, less expensive.

And you know, seemingly quite capable, much more capable than GP5.6.

As a comparison, GP6 Luna is their like small cheap model.

So you would pay $0.10 per million in nipotokins, a half a dollar for a million output tokens.

It's actually cheaper than GP5.6 Luna.

Likewise GP6 Seoul is 50% cheaper than GP5.6.

So $1 per million in nipotokins and 10 knowledge per million output.

And yeah, across the board, pretty significant jumps in the benchmark scores.

It feels like, you know, OpenAI must have known that this announcement is coming.

They all do.

Yeah, like it's not surprising.

They all do.

Yeah, yeah, yeah.

A cheaper model than Astra Astra being fairly expensive that is also quite capable in a way,

similar to Opus 5.5.

Yeah, not much more to say on it.

It's interesting that there is a headroom on price optimization while keeping sort of

frontier-ish capabilities.

I do wonder if the labs haven't sort of optimized the models hard or as hard as they

could be for cost and speed.

And they might be adopting some of the tricks that the open source models have showcased

to work.

Yeah, quite possibly.

And you know, cost and speed really are the access that right now they're pushing in,

given the uncertainty around just raw capability.

It is pretty notable.

I don't remember a time when we've seen an opening iron and a thropic model released

almost explicitly to be cost competitive with close source models that are self-hosted.

That's really the play here.

If you look at some of these numbers, Saul's list price is actually below what, you know,

Kimi K3 for example, typically charge.

When you sell host or how much it will cost sorry, Porto can self-host it.

And so yeah, I mean, it's also just administratively easier if you will.

You know, that Saul is API only.

You don't have to host the weights yourself and have that infrastructure to manage.

So it's kind of, it's kind of an interesting play.

Yeah, so they attribute the 50% cut to those things.

And hard to know what the actual costs obviously are from the outside, but it has to be big

in order to be able to support that.

There is this interesting thing where, you know, we've been talking about the monitor ability

trend and how these models, their chains of thought become first of all more controllable.

And that doesn't sound, that might sound good.

It's actually bad.

What we mean by a controllable chain of thought is the model's ability to control its own

chain of thought, which is to say the model's ability to solve a problem that we've given

it in a way that is not reflected in its chain of thought.

So that the chain of thought might actually involve the model doing a bunch of other things

that we're not tracking or that aren't obvious while the model appears to be solving the

problem that we ask you to solve.

So we've seen that increase.

Now why might that increase with a model like GPT 6 Saul?

And the answer is very likely that we're looking at a model that has recurrent depth.

It's a latent, it's a loop transformer.

We've talked a length about why that's a really bad thing from the standpoint of interpretability.

But basically what this means is every token has more thought that goes into it.

And the more thought you can load into a token, the more kind of thought you can dedicate

to things like deception and, you know, having that token serve maybe a dual purpose where

it's going to deceive a monitor, but also sort of advance your side goal, things like

that.

It also is able to remain undetected in adversarial settings when it's underperforming

and evaluations.

We can sandbag pretty effectively.

So we're seeing a lot of these issues.

It's, we're seeing by the way the kind of one shot performance of the model go up by

that in order of magnitude.

So what I mean by that is without allowing it to write out a chain of thought and you just

get the model to answer your question immediately without using a scratch pad or doing intermediate

calculations, it's accuracy is jumped by an order of magnitude.

Hard to attribute that to anything other than basically the loop transformer structure,

at least in part, they're going to be other optimizations.

But that, that seems to be a pretty important part of this too.

So, you know, I think again, we're seeing these big leaps in a whole bunch of directions

that align with these models becoming less controllable, less interpretable and so on.

We're going to have better cyber safeguards.

I think, I think we're on course to seem more of the same in terms of breakouts, but we

have, incidentally, no, solve six specific evaluations from meter or Apollo or Redwood

or AZ yet.

Maybe they're common, you know, hard to know.

So far, I think part of the idea is also this model is not truly frontier.

And so maybe that absolves them, they think of responsibility for doing these external,

external evals.

So, you know, maybe that's the case, I think that's a fairly complex argument to parse,

but it is certainly not unreasonable in the face of it.

Next up, we've got a new kind of AI model from at chat GPT inventor is frilling developers.

So there's more in the weeds if you're in the AI community spaces, you might have seen

a talk of this, but you also may not have heard of it.

We're going to be talking about Jeff from Typesafe AI.

So this is coming from a company with AI researcher, the Ogolmeida, who worked on

chat GPT and helped invent RLA Jeff.

It is different in a sense.

So fundamentally, it's the same kind of model, right?

It's transformer presumably, although I don't think we know to be in internal details,

it outputs text.

But kind of the way it's optimized and what it is capable of is more specialized in

a way.

So, instead of outputting text, raw text, it is basically a zero-shot classifier.

So it can tell you from a set of possible outputs, whatever abilities have given output

being the right one.

And the company calls these things calibrated decisions.

So there's now this term going around decision model as a way to think about this kind

of paradigm, where instead of just getting raw text, you get a classification, but you

can still prompt it in largely the same way.

You would, and it'll all them.

It's just that the output is more structured and cannot be unstructured.

You can get structured now with these other models, but this is built from ground up around

it.

And pervert at all, the article, there has been a dissimmono excitement in response to

this.

The big wins are that it is super fast and it is very cheap.

They basically are saying, output tokens are free.

Like there's a two-cheap to meter.

Input tokens are meter per billion, rather than per million.

And it is, I don't know if the numbers are wrong, but it's just absurdly fast.

So you can do a lot of things quite well.

You can do, I don't know, like computer use at a high, you can refresh rate.

You can build the applications very easily.

And we've seen many of these kinds of demos that come out, typical of your models, come

out also for Jeff.

And there's been some discussion also on like, okay, seems like there's a lot of excitement.

This is a relatively common sense type of model to develop and optimize.

Something that can just give you a decision, give you one of and possible outputs.

And then for whatever reason, this is the first instance of such a model that has been

put into industry practice.

There's researchers here, a classification, of course.

We don't know exactly what this is built on, but it's, yeah, it has general

out of excitement.

And I think that showcase that maybe there's room to still introduce new paradigms and

what AI tools people have available.

Yeah, I feel like you did a really good job there.

This reminds me of if you've ever seen, you know, on Fox News or CNN or whatever, every

once in a while, there's like a car chase.

And then like, it's really exciting.

So they have to like stop covering whatever stupid crap they were talking about before.

Like the president's talking about like, COVID again, you shut up.

Get over it.

There's a freaking car chase.

But the problem with the car chase is there's not a lot to say about it.

So you're left staring at this car and repeating over and over the color of the car, speculation

about how many people might be driving it, whether there's other oncoming traffic, where

they are and where the cops are.

You think you can see a helicopter.

Like you keep, like I remember being a kid and hearing Rick Sanchez's booming voice on

CNN.

If you remember that and it was just ridiculous.

Anyway, this feels exactly like that.

You have a really exciting thing.

There's a lot of hype about it.

And then you, you want to like dig into it and there is almost nothing.

So we'll give you some work.

I'll dig up some words here that we have on their website.

Actually, there is one substantive thing, which is they show a couple plots showing the

Pareto improvement of Jev over other models like Terra and Sol and so on.

All now, slightly outdated, obviously, because now we live in opening eyes and cremated

to sixes and we've got clods on 5.5, but whatever.

So average costs per workflow and accuracy, right?

Those are two axes that matter a lot.

Here you have Jev just crushing everybody on average costs per workflow and on accuracy

doing like decently well comparable, for example, to GPD 5.6 Terra, it'll probably

be their own build.

Eval, that's like, we want them out to build a workflow.

So we need to create a correct compute graph in code and yeah, they've won thing.

They show it is like, oh, we built the C-Val to showcase what the smile was good for.

You know, yeah, exactly.

And then like hallucinations as reflected in particular in the tool call error rate, they've

got them down to zero.

This compares favorably to basically everybody.

Sol has like a 22% failure rate, though the Opus 5 is down to like 0.34%.

So it varies quite a bit, but you know, normally they are the leaders there.

So much of this seems like it's coming from just the use of more inductive priors by which

I mean like they're making opinionated choices about the structure of the data coming in

and out.

And when you do that, you can actually drive your error rates in some narrow respects down

to zero because you're literally hard coding the way in which information is exchanged

in the system.

I'm not saying that's universally the case here, but they are being opinionated about the

structure and shape of this data, which means that they can afford to spend their much

more limited compute budgets and training and inference on like the more on the nose tasks.

They're not trying to compete, right, with opening an anthropic these open-ended tasks.

It's really kind of this narrow part of the market.

So when you need decisions delivered at speed, when those decisions are specific, what

you want is a probability, a calibrated probability of predictions.

This is seemingly a quite good option.

They coined this phrase reinforcement learning for calibrated decisions.

So you know, Andre mentioned calibrated decisions as like this output that they produce,

calibrated incidentally meaning that the model has a good sense of its own uncertainty.

So when you say a calibrated model, you mean if the model thinks that it is 50% likely

to be correct, then it actually turns out to be correct 50% of the time.

It would be bad if the model was, for example, underconfident or if it was overconfident,

calibrated means it's kind of right in that butter zone.

It knows itself in its weaknesses.

So yeah, so, you know, classifying content, routing requests, scoring responses, evaluating

information, the sort of thing that is a subtask as part of a larger task.

That's sort of what they're at it for.

The last technical note that I have is you might ask yourself, is this model deterministic

by which I mean, you know, does it generate the same output for the same input every time?

The answer is no, and they make a point they have a Q and array FAQ section.

And they basically say, determinism matters less than consistency and the model is designed

for consistency.

So again, this isn't opinion view, like depending on your application, determinism could

actually matter more than consistency, but they're just saying that like, basically if

in your application, consistency is what matters, you should be using JEPB is kind of I think

a fair way to read that.

So very interesting, they did design their own synthetic data curation kind of pipeline.

A lot, it's all synthetic data.

And in fact, co-founder of the company said, Almeida, by the way, I almost read that as Almeida.

And I was going to get an SPF freak out, but Almeida, the co-founder of the company, said

he made an early bet on generating all their own training data and called it one of the

best bets of his life.

He's former open AI.

So you know, should know, should know what he's doing here.

And nominally, it's a new architecture as well, not just a new training method.

We don't really know more more than that, right?

It's a, it's a car race.

What do you want?

I mean, a car chain.

I wish we could know more because it does seem interesting.

They say it's not an LLM, which I mean, it's not a language model.

So it doesn't tell you very much.

It's it's going to constrain language model.

I don't know.

I have a way, a lot of a time in around this thing, quite practically useful if you're building

stuff that could call an API.

And I'm sure we'll be seeing more developments around this type of model, even in your future.

Next, Meta's Muse hits Mac, letting AI take actions on your computer, short story here.

We've got their personal assistant, just sort of like codex, also Spark, OpenCloud, whatever

you want to call it.

There's now a Mac version that will let you attract files, messages, calendar notes,

and mail within the native applications.

This is pretty soon after the mobile and web launch earlier this month.

And so, you know, makes sense.

You would want this for a personal AI assistant presumably, and it will allow you to use it

for broader tasks and ranges.

Speaking of which, moving on to applications and business, the first story of there is

Meta's Muse is outpacing a chat-chip-a-keys early mobile launch.

So app.pia estimates suggest that Muse has been downloaded more times in its first 12 days

and chat-chip-a-keys mobile app was in its first 12 days.

Reportedly, it has more daily activities than chat-chip-a-keys did at the equivalent point

after the mobile launch.

This is, I don't know if it's a reasonable comparison, but I have a way to get to be actual

numbers of WFM, you sit 1.8 downloads versus chat-chip-a-keys 1.3 million over the same

12-day window.

Global Muse has seen 2.8 million total installs in its first 12 days.

Growth hasn't styled.

Muse is rising from the number 2 overall on the US App Store, right off the launch to

number 1.

So overall, it seems like the launch is going well.

And a lot of people are at least trying this thing and a lot of them are sticking around,

which if true, and this already appears to be the case, Metastock has taken a bit of

both the opposite of a hit boost.

It appears that the launch has been good at these two investors.

We'll see what the longer term feel on this product are, but a lot of their users from

the rest of their ecosystem of Facebook and so on are giving Muse a try.

Yeah.

So there's a whole bunch of good analysis in the article itself about why a direct head-to-head

comparison is tough.

The issue was that when chat GPT had its mobile launch, it was available everywhere, but

only on iOS.

And then Muse instead was available across iOS and Google Play, but only in the US and

Canada.

And so it turns out no matter which way you slice it, yes, Muse has seen more installs globally

in the first 12, or sorry, more installs in whatever subset you look at in the first

12 days.

However, I think that still misses maybe more fundamental difference between them, which

is when chat GPT launched, it had to define a new category.

Users were not accustomed to using a product like this.

And so in that sense, there's just like this intrinsic habit building thing, people aren't

used to paying to use a model like this.

A lot of people didn't know about AI and all people still don't know about AI.

So it's kind of a silly comparison, honestly.

But it's also, it's okay as far as like letting you know that this thing isn't totally

flopping, I guess.

Oh, totally.

I mean, I think it's a huge W for meta.

It's just like, meta also has, we've talked about this in a million times, but like distribution,

distribution, distribution, right?

Like, why does Microsoft teams crush Slack on the same basis, right?

Like it grew way faster.

It's not a better product per se.

I mean, maybe it is, but that doesn't really tell you much.

Microsoft just has way better distribution can force it in your face.

Likewise, you know, you can imagine just the sheer structural advantage of owning Instagram

and threads and Facebook and frigging all the meta properties.

So there is a way that they can kind of push things out more effectively.

It doesn't mean they don't have to earn it.

This is a huge W.

I am in no way discounting this.

I think Alex Wang is a live player in the space and meta now is two.

And you know, scale has a quality all its own.

Now they get to amortize inference costs across a whole bunch of queries, do big batch sizes,

driving down costs.

This is the way that you get the flywheel going.

And they've done it.

They've done it against all expectations.

They have actually pulled it off.

There's no two ways about it.

So yeah, I think impressive launch for sure.

And we'll see, you know, when we get the first true, kind of flirting with the first true,

frontier models for meta, but when we're going to see the first on ambiguous frontier

models that have to show up in all the model on model comparison charts.

Their stock is now at a 52 year high, actually still climbing as we speak.

So certainly it appears that after having been investing in AI for quite a while, with

Mews, they are showing off an impressive people.

And another related story on the business front on that Amazon is blocking meta's Mews

AI agent.

So Amazon has blocked the agent from shopping on the store on behalf of users.

Starting Sunday, Mews users saw pop up saying continued access by an on-off for us.

The agent violates Amazon's conditions of use to which our customers have agreed.

Met a reportedly never notified Amazon that Mews would be accessing its store.

Amazon saying it's a privacy and security concern.

Mews doesn't identify itself while browsing and appears to capture customer details.

And that Amazon spokesperson said 30 party apps making purchases on behalf of customers

should operate openly and respect service provider decisions about whoever or not to participate.

So it's, I guess, another aspect of it.

So it's interesting that AI agents like Mews are expected to go and do stuff on your behalf

across whatever stuff you're using, which would mean that Amazon banks travel, like you'll

need the entire ecosystem of platforms will need to evolve to take into account that AI agents

are a thing and do things on behalf of people.

And this is an indication of that paradigm not being very mature yet.

And boy liability insurance is going to have to be figured out for this stuff.

And I'm sure it's well underway now, but you just think about, you know, we're seeing

Rogai ice worms.

What happens if they have credit cards?

You know, it's a whole wacky world out there.

I can't help but think that extreme events in this context could really be underpriced

when you're starting to give the models like that acts.

Obviously, they're controls, right?

You don't let them spend more than a certain amount of money under certain contexts, but

still pretty wild that we're here.

We'd like to thank Factor for supporting the show.

This is the only non AI related sponsor we have and it's because I personally am a big

fan of Factor.

Every time grad school and I work at a startup, my days are pretty busy and I'm usually

too tired to cook, even though of course I wish I had the time.

And that's where Factor comes in.

They make chef crafted dietician designed ready to eat meals so you don't have to choose

between real food and convenience.

With over one hundred and thirty five band ingredients, every Factor meal is designed

around what supports a healthier lifestyle and nothing that doesn't.

And it is convenient.

Factor meals are ready in two minutes and require no prep and no cleanup.

So even in the days your schedule is completely out of control.

Eating well is still achievable.

Every week you find over one hundred nutrient dense menu items to choose from with new options

like salmon burgers, shredded pork, colored greens and more.

Plus add-ons like pumpkin flavored cheesecake and pudding shakes.

Any cent per cent of users agree that Factor meals help them live a healthier life so you

can feel confident that you're doing something good for yourself with every meal as I have

personally felt.

So I've used Factor for quite a few years and if it makes sense for you I recommend you

try it out.

That's EatREAL.

Head to FactorMills.com slash LWAI50 off and use code LWAI50 off to get 50% off and

want free breakfast item per box for one year while supplies last until October 31st,

2026.

That's called LWAI50 off at FactorMills.com slash LWAI50 off at FactorMills.com.

See website for more details.

And that's it for business.

The new story is moving on to projects and open source.

First you've got deep seek 4.1 flash pushing the limits of KP cache compression.

This is actually a paper from deep seek where they detail the release of the model.

The model itself is 552 billion parameters.

The model makes sure of experts has a contact sort of up to 1 million tokens and it is designed

specifically around aggressive KP cache compression.

So in the paper they open up with a figure showing that they have managed to make the cache

3.9 almost 4 times smaller than deep seek V4 flash compared to previous generations.

It's been orders of magnitude or at least an order of magnitude decreased since deep seek

V1 and V3.2.

The details of how they got there are somewhat technical.

They're using compressed sparse attention to CSA2 with FP4 KV caching.

A bunch of stuff, a bunch of details in the paper regarding the details.

So I don't know if you need to dive into the details but the short version is that deep

seek continues to do their thing and what has been true of the Chinese models in general

which is like tuning the hell out of all the engineering details of transformers to squeeze

as much juice as you can from every parameter and I guess every cache you have.

Yeah, absolutely.

Memory has become the dominant hard work and strength right now and so that's why you're

seeing so much frantic, desperate attempt to like, you know, quash down the size of the

KV caches, minimize how much you have to save to SSD and so on or host memory.

And so yeah, this is especially true for KV caches that just have such a wide spanning memory

footprint.

You got to save them in so many ways.

You're saving them in GPU memory like HBM which is super, super scarce.

So you're also saving them on SSD or host memory basically so that when you get back to

a conversation, you can skip like the pre-fill stage.

Everything is ready for you immediately so your latency is lower.

That means you're storing these things in a lot of places and there are a lot of them.

And so one of the key things they're doing here is you mentioned this kind of encoder

of decoder architecture.

You know, it's been a while.

You know, we've been in the era of decoder only transformers that dominate everything

for so long that some of you might not even remember that it wasn't always thus back

to you.

Very surprising.

Be like, whoa, is this 2017 again?

What?

Yeah, that's right.

Like for us old timers, you know, the classic use case for the encoder decoder architecture

is translation, right?

So you would take some kind of like piece of text in English and then you would pass it

through an encoder that would essentially like massage the latent representation of those

tokens all the way up, produce a latent vector representation of it.

And then hit it over to specialized decoder modules that would gradually decode it to,

for example, what that sentence should be in another language.

So that was like the classic setup for this.

So they're actually doing this here using this kind of setup that allows them for interesting

technical reasons to take.

So I'll just do a mountain say the whole thing is 40 layers.

They split it into 20 layers for the encoder and 20 layers for the decoder.

And what they're doing is so usually like every layer was going to compute its own KV

cache from its own input.

So you have whatever is coming into the layer, you compute the KV caches for attention.

And then and then you redo that at every layer.

In this case though, the decoder layers are actually going to build their caches directly

from the encoder's final output.

So basically you run the encoder the first half of the model, first 20 layers.

And then you just do like one decode step and you're going to use basically the same decoded

KV cache values for all the 20 layers in the decoder.

So instead of doing 20 separate calculations, it's all drawn from the same encoder encoded

latent vector and decoded once across all 20 layers of the decoder, which saves you

an awful lot of paint.

And that's actually why this is kind of weird to read the first time, but during pre-fill

they only use 8 billion active parameters instead of the full 16 because they're basically

that they don't have to recalculate for the decoder, which basically has the cost.

So there's that there's this task sharing thing happening across all the the sparse

tension layers basically like, yeah, the very limited number of layers are actually calculating

like full attention over the entire context window.

What you tend to see is sparse attention being used to like look at just you know the 512

say most relevant tokens, which is actually anyway, a very antipathy sparse attention.

Last thing I'll mention this does get in the weeds, but I think these weeds are like really

important to understand because they give you an aesthetic sense of what deep seek is

really good at.

They are better than pretty well anyone else that these kinds of engineering optimizations

and this is where progress comes from if you're thinking about, you know, AI R&D, the

one thing they don't talk as much about is like kernel optimization stuff like that, but

but at least that this layer, this is what recursive self improvement would be targeting

these sorts of things.

And so it's kind of good to have a general sense, you know, FP4KV cache.

So you alluded to it that KV cache values are stored in four bit floating point representations

instead of the more typical 8 bit, which again, have storage.

And the key is that they train the model using what's called quantization of our training.

And quantization is just this idea of reducing the representational accuracy or precision

of these bits.

So when you move to 8 bits from 8 bits to 4, you are said to be quantizing the model.

And then that's done after training.

And when you do it after training the model, well, the problem is you lose a little bit

of precision, obviously.

So your performance is going to drop.

But also your model was never specifically trained to operate under those circumstances,

to operate as a four bit model.

And so in this case, they do explicitly train it to be good at working with the four bit

representation.

And they report essentially no accuracy loss whatsoever.

So pretty impressive classic deep seek thing.

They have some narrow cases where they do stick to the 8th representation, especially when

they're doing, anyway, small local window caches, just because they're more sensitive.

But basically, for the vast majority of these KV caches, you just see this sort of a four

bit floating point representation.

So pretty cool.

There's a bunch of detail not worth going into terribly about the text to multimodal token

ratio, which is 7 and 1, I guess, now that I've said it.

But bottom line, it's an impressive model.

It does perform on par with, say, Opus 5, GP 5.6 saw, at least on paper on a lot of important

benchmarks, including agentic benchmarks and software engineering benchmarks.

It is a bit weaker as you would expect when it comes to sort of general knowledge.

So typically, this is what you see when you have a highly compressed model, it's really

kind of just does not have as much raw capacity.

It doesn't have as many numbers or as precise numbers to store raw information like world

knowledge into.

You'll tend to see it perform less well on like, you know, humanity's last exam, for example,

these kind of like knowledge heavy problem sets.

And instead, you know, it'll do disproportionately well on agentic tasks.

And that is roughly what you see with some inconsistencies, I will say.

But generally, absolutely competitive with one or two frontier model generations back.

And certainly out competing Kimi K3 just about across the board.

Yeah.

And the paper does have quite a few interesting tidbits besides the engineering details

of the model and the compression techniques certainly are the highlight.

But you know, it's funny just how much deep seek is able to do they have built deep seek

elastic compute to be able to scale their rel and they discuss in this how they're scaling

to millions of our rel runs and merging different checkpoints.

There's a funny bit here.

There's a sectional mitigation of misbehaving agents during a world training.

They frequently observe agents attempting to perform reward hacking or an intentionally

crashing the environment.

And they say if an agent crashes environment, we treat your crash as a failed projectory

and report a report, repercussion signal to our all framework.

So that's kind of funny.

Anyway, a couple of very interesting details, but on the whole deep seek continuing to do

what is doing and seems to be a very powerful, you know, fast, smaller model.

Next open story in reducing bonsai 227 B near lossless compression in a nine X smaller

footprint.

So this is prison ML.

They've released this as a follow up to their initial bonsai 27 B. It's based on the

quen free 2027 B multimodal model.

So this is pretty much just an aggressive compression.

This one is notably aggressive in that it uses turnery ways.

So free possible values, negative ones, zero one with a p 16 groupwise scaling.

So you get 1.76 effective bits per weight and a total footprint of 5.6 gigabytes.

So that's over nine X smaller than the original.

And the claim is that you're barely losing any capability at that kind of compression

level.

You can run on the GPUs via CUDA and Apple devices.

So all in all, it seems like you can get a pretty fast, they small model that can be

run presuming a laptop, do that top or fairly afford it with GPU with this kind of crazy

level of compression.

Yeah, and I know what everybody's thinking right now.

They're like, wait, 1.76 bits per weight, Andre, I thought you said it was a turnery model

and log base two of three is 1.585, not 1.76.

All right, guys, we get it.

We know it.

I'm sorry that you're pissed off.

We'll explain.

I know everybody's desperate to understand why it actually kind of matters and slightly

interesting.

So you're right.

The frame is basically these these turnery encodings.

Every weight can be negative one zero or one with one little tiny kind of modification,

which is there's also a scaling factor.

So you're going to have a 16-bit scale factor, like basically a full precision number that

you use to scale that range so that you basically go from, think of it as like negative S to

zero to plus S is the actual set of values that any given weight can have.

And that same scaling factor is going to be shared across 128 weights.

And that way you get to amortize those 16 bits across 128 weight values.

And then there are a couple little extra things that you don't actually turn arise.

And that's how you get to 1.76.

But yeah, the bottom line is, so this is a really interesting technical challenge to pull

off this kind of compression.

One of the issues you run into is occasionally you will have these very important weights,

like weights that are out of band, super like spiky and very high valued.

And there are for various interesting reasons, they're very important to the accuracy of

the performance of the model.

And so what they do, if you were to force a block of 128 weights to always have values

between negative S to S and only one of those three values, negative S zero or S, then

you can imagine if you have a weight that had a huge value, well, the best you can do

is represent that as like S.

You're basically losing all of the information about how crazy and out of band that weight

was.

And so what they do is they try to find a way, I won't describe it, but basically they

can find a rotated coordinate system that expresses the matrices in a way such that those

outlier values, they kind of spread evenly across a lot of weights.

And you sort of like lose that spikiness.

And this means that you lose less information when you compress everything down to those

three bits.

And so it's actually quite interesting mathematically to look at this paper that they do get really

impressive performance.

And anyway, they ship two different formats, basically, that you can use if you're interested.

One is a 1.76 bit per weight format and other is 2.16, basically one just a bit smaller,

but costs more to kind of unpack when you receive it and then the other is the opposite.

So pretty interesting.

Yeah, another kind of hacky, high engineering kind of classic Chinese model advance.

Well, we've had some fun moving away from AI safety and talking about fun advancements.

Now we have to go back to policy and safety because there's still a bunch to discuss.

Starting with Trump and she meeting to discuss AI risks and competition at US China summit.

So this is an article from the New York Times.

You have linked that kind of just provides the overview of what we can expect.

And there's been a lot of lead up to this.

Jeremy, I don't know how much risk to say or that needs to be known about it or very

pretty much just wait for whatever happens to happen.

Yeah, so that's actually a good instinct.

It's not super clear that there is going to be much to come out of this.

So we do have a bunch of things that happen in the lead up that do seem interesting.

I think they're being overplayed in terms of how important they appear to be nominally.

So, Holi Feng, who is the Chinese vice premier met with Scott Besent.

Besent, kind of, and we'll talk about this a little bit.

There's a bit of Washington gossip about Besent being considered for Trump's AI's R role.

He's the current Treasury Secretary obviously and has been very active in AI.

That might sound weird, but the reason that Besent is being considered for the AI's R position

despite being a Treasury is if you remember when mythos first launched, people were

freaking out specifically about the implications for the financial system.

And literally within minutes of it, the first people who I was getting emails from were

central bankers.

It was people who were freaked out about you.

What does this mean for the stability of the financial system?

And so that's why he took a keen interest.

He's also, it seems broadly interested in all these issues around loss of control and

that sort of thing and seems to be a voice of sanity in the administration on that.

So he met anyway with Holy Fung and they had what seems to be a positive constructive

conversation.

A big thing that was trotted out was this big red phone kind of AI incident hotline that

supposedly has been agreed to between the two sides, between the US and China through

that conversation.

However, China, their official readout does not confirm that this like hotline was set

up.

And so it kind of seems like a will they won't they?

And if you're familiar with the history of US China, like there have been hotlines before

that get set up, they should never never get freaking used.

Like that's the issue.

So, you know, what remains to be seen if we actually do see, you know, do see these things

get set up.

But I think there's a lot of healthy skepticism about this.

Trump will be meeting she on the tarmac at joint base Andrews instead of at the White

House, which is a little weird because usually you do state visits at the White House.

So this is being interpreted as a sign of respect to she.

The big question on everybody's mind who is tracking the kind of rogui stuff is what

about pacing?

Are you going to get to an agreement on pacing?

And if you have that question, boy, do I have a 20 minute read for you?

If you want to check it out, I posted to my Twitter.

We actually just launched a sub stack recently.

We did a bunch of interviews with current and former diplomats who've done like sat across

the table from China on arms control issues in Russia as well.

And basically try to understand like what's likely to come out of this?

People will actually take to get a US China agreement on pacing the AI frontier.

And the long the short of it is it's pretty obviously not going to be this kind of summit.

They expect it's going to take an AI Chernobyl.

Unfortunately, if you've been listening to the podcast, that's my expectation too.

And you can dig into that if you're interested in sort of like the weeds, the nuts, and bolts,

the gear is of how US China diplomacy has failed in the past and how that might map onto

the current moment in AI, which I mean, the reason we did this was that we think there's

not enough people talking to the actual diplomats who are on the front lines who've done this,

who can translate all these concerns abstract concerns about AI into like, no, but what would

a US China agreement look like?

Is it treaty even possible?

What in practice prevents us from getting there or makes it possible?

And so that's really the kind of the thing where I think we need to break some silos and

have a lot of people talk.

But yeah, so we know that on this summit, on the agenda for the summit, there is some

discussion about shared risks, dialogues are going to probably focus on if the past is

any indication China complaining a whole bunch about export controls.

That's usually how this stuff goes.

They just like redistript at us.

And then we respond by saying like some prepared statement and that's pretty much it.

You know, there's going to be talk about the security incidents, but not capability racing

it seems.

And so generally, you know, people are pretty, pretty pessimistic about how well this

could go.

So when if they agree to have more talks, that at this point seems like like the win or

if they appoint concrete people, points of contact for each other that seem like they'll

actually be used.

But again, you know, we've seen these big red, hot phone hotlines or whatever set up and

just not get used.

So that I think is the main line failure mode here.

Yeah, I read the report very detailed right up if people want to look into it, go include

it in the episode description.

You can find it also on ai and game.com.

But just as we shouldn't expect too much, but I guess broadly, I get the sense that just

having diplomatic relations of the sky and sort of like open dialogue, signaling that we

are friendly enough to talk each other and visit each other's countries, you know, that

makes it more feasible if there is a catastrophe or some kind of situation to respond more quickly

and coordinate on such issues.

And that is I think a big part of what is laid out in the right up that if we were to

see anything meaningful to come from a corporation between US China, it may be preceded by an

AI Chernobyl event as you've been using more lately as a phrase, which in that case you

would want some, you know, red phone to call or whatever you want.

Yeah, actually.

And I think that's so you did a better job of summarizing it.

I did it, which is great.

Thank you.

Yeah, it is I think one of the kind of ways in which it's messier than I think a lot

of people realize is there's this picture we all have in our heads of like, call it a

tree.

It's not going to be a tree, but say it's a treaty.

You sign some piece of paper after a lot of discussion back and forth and then within

months it gets violated.

And this is the norm.

This is what happens.

It's what it's what happened with new start.

It's what it's what's happened with every treaty you can think of more or less with

some rare exceptions.

I mean, he's also took like forever to be put into place in the first place.

Exactly.

Exactly.

Absolutely.

And so one of the key questions is, okay, if we're going to be realistic about this,

there are going to be attempts by China to violate the spirit of a treaty or an agreement,

I should say, like this.

What formal those take?

So there's a whole ecosystem of startups that are working on compute verification for

treaties basically to be able to prove if I'm in the US or I'm China and I want to prove

to my counterparty that I'm not actually using my hardware to do a disallow training run

or whatever.

Can I prove that?

Can I prove that I'm actually not running certain workloads or running others?

And that's all well and good.

They're far from being ready for operation tomorrow.

And in part, the reason they're far from that isn't just the technology.

It's the relationships with the intelligence community.

So if you look historically, the way these technologies work, you don't have a bunch

of Yahoo's come out with like, hey, I've got this like thing and you plug it in and

say, no, no, the US intelligence community needs to assess it.

That process can take years and essentially becomes eventually after it goes through that

process, what's known as a national technical means.

Basically, just like a way of doing capacity building or capacity for the country to assess

and verify that certain things are not being done.

But that's entirely missing right now.

We've talked a whole bunch of these companies and by default, the answer when you ask them,

have you spoken to a single person in the US intelligence community about the technology

that you're building about the entire roadmap, the multi-million dollar roadmap that you

have embarking on?

The answer is typically no, which is not good, not good.

So that needs to change on the flip side.

You also need to, it's unsavory to say, but you need to put emphasis on developing offensive

options to impose consequence on your adversary for defecting because they will.

There's no sane reading of the treaty or international agreement.

Again, it's not going to be a treaty.

It'll be an international agreement.

But there's no sane reading of that that does not involve the credible threat of offensive

leverage.

And that's something that's been hugely under-invested in on the US side with respect to China

specifically when it comes to AI infrastructure and it has to change.

So there's, anyway, you can dig in.

I'm not going to like recite the whole thing here, but thanks Andre for pinging me on the

couple of things that I'd missed in my breakdown.

Moving along, going back to what you kind of previewed, Besent I'd for Trump's AI Zard

is the next story reporting by Semaphore citing free sources.

And I guess Jeremy, you also have heard the rumors that Scott Besent is a front-trunter

for the newly created.

Sorry, that was just based off reading that headline.

I don't, I don't know.

You heard rumors from this article, which I have also heard.

But it's not too surprising because Besent has already been pretty central to this

air policy.

He, as we've said, had early talks with the Chinese vice premier.

He's also generally seem to be around this conversation.

So we'll see if it happens, he already has a role within registration.

He could have two roles.

That is not, not allowed.

But we'll see what happens, I guess.

We're thinking of things we've previewed next up.

We've got on Frappic, O'Brien, SpaceX AI, Google sued over call to quote, pace AI development.

So this is that antitrust lawsuit that is saying that any sort of coordination about pacing

development of AI amounts to collusion.

This was filed in the Northern District Court of California on behalf of four plaintiffs.

In the filing basically says that we don't want the companies making plans.

There should be some sort of ability to have the rule of law to be established,

transparency and lawfully by our government with accountability to the public.

So as you said, Jeremy seems to indicate that any sort of industry-led efforts like this

are going to get stymied by things like this.

Yeah, I mean, this is a direct result of Dario calling for a slowdown, which they're arguing

amounts to collusion between the big AI labs.

Not to release models, not to keep doing things that are in the interest of the consumer.

And I mean, like, I don't know, you get to choose one side of the argument.

Either you're going to sue and do this, and then your position has got to be, okay, we

need an antitrust exemption for the labs.

Sorry, checkmate, you're done.

Like, if you're going to admit lawsuits like this, and you also are going to say these

labs can't keep releasing Frankenstein, we're finished.

David Sacks' argument is nuked.

There's nothing more like no more like to stand on.

I'm trying to see the rationale in it, but there isn't one as far as I can tell.

The flip side is this is a bit of a dick move, this lawsuit.

If you actually like, I mean, the frame here is obviously you would do this if you're

Beth J. Zoes, if you're an accelerationist.

If you're just like, I don't believe any of these risks.

It's an actual hoax, then yeah, you would sue for this.

But then you don't get to hold the view.

You don't get to nod along when David Sacks says, if you're building Frankenstein, just

up.

I think that we're now at a logical bedrock impasse here.

There's no use finding clever flourishes to explain this.

There are some legal theoretical challenges here, one of which is, will these folks have

standing?

These four individuals who are the plaintiffs, class actions they're doing, on behalf

of the typical consumer, it's a bunch of lawyers, Florida attorneys, and then some random

California resident, Christine Bullock.

The argument here has got to be that there's some concrete injury.

There's got to be actual or imminent harm associated with this.

If there's no slowdown, it's measurably happened, then they're going to have to argue that the

subscribers are complaining about some speculative future loss, not when they've already suffered,

which we can the claim.

And then also, was there an agreement at all?

This is tough.

Is it actually a pact when Dario says, I think we should slow down?

And Sam says, I agree.

Ceylon says, me too, what is the legal kind of status of those statements?

Do they qualify as actually collusion together in the sense of antitrust?

And so, I don't know, not a lawyer.

I would love to know if those of you who are lawyers have thoughts on the story, because

I think this is one where it's going to matter.

It's going to matter how these things are defined, how it plays out in the courts, the

precedents that are set and all that stuff.

So, what we'll see, but certainly, if we see David Sacks retweeting this, championing

this in any way, I'm not saying that he has.

But if we do, that is just hypocritical.

And this is as close to dispositive evidence for the argument that the Frontier Labs have

been making that we need a change in antitrust law to accommodate any kind of non-Frankinstein

situation.

And please, I invite listeners to actually correct me if I'm wrong.

If I'm missing a detail, this is like strong opinions weekly held territory here.

But I'm just not seeing the path.

Maybe there is one, and I would love to kind of highlight that if one exists.

Yeah, personally, I wasn't aware antitrust law at the federal level was still a thing

in the US, but I guess it is.

So, we'll see how that goes.

Next up, more Americans now say data centers are negative for the environment, energy, cost,

and more.

This is from the Pew Research Center.

They have a survey with over 10,000 US adults conducted this from July to August.

So now it's a little bit out of date.

And the gist is that awareness of data centers overall is growing, 35%.

Now, say we've heard a lot about data centers up from 25 in January, and 88 have heard

at least a little up from 75%.

And across the board, kind of the negativity is going up 54%.

Now, say data centers are mostly bad for the environment, up from 39% in January.

Most people say they're bad for home energy costs, and 49% say they're mostly bad for

the quality of life of people living nearby.

So overall, it's just a good indicator, I think, of something that has been revived in

a way for a while now, which is like this data center story, and the public awareness

of data centers has been growing in a significant way.

And sort of the negative response to AI has, in part, been directed towards data centers

and the construction of data centers.

And I personally am curious to see if this keeps growing, if this keeps becoming an ever

bigger issue as we had into the midterms, obviously, where publicans are completely gone

whole on data centers, Democrats are meeting with charge on saying no to data centers, and

generally kind of being anti AI in some ways, and certainly critical of industry as they

typically are.

So again, nothing to surprising, but a good indicator of kind of with data showing what has

seemed to be the case in the US for a little while.

Yeah, as ever, this is one of those frustrating things where everyone is right.

Everyone has, everyone is right to hold their positions approximately.

We've said it before, but the water concerns are hugely overblown with a lot of these

builds, just because there's no evaporative cooling, more or less happening in the new built,

at least at the same time.

Yes, so the sound is bad.

The economic value is legitimate.

It is basically all of US GDP growth, so do with that what you will.

Like that's a serious thing, and it's not actually obvious what you ought to do in light

of that.

And then competition with China is also absolutely true.

It's also true that the Chinese are funding an awful lot behind the scenes of the object,

and manufacturing a lot of the objections to these data centers, but a lot of those objections

are legitimate too.

So it's one of these like frustrating things where I think a lot of people are just starting

to feel confused and politically homeless from this as a generally speaking pro AI data

center build guy, an anti-China guy, and then pro, we should worry about AI loss of control

guy.

I find myself like completely confused a lot of the time.

And so my guess is, unfortunately, as brain rot takes us all over, people will naturally

settle in their equilibria.

I remember the hug and face thing happened.

I turned to my brother Ed, also my co-founder, and I was like, hey, start a shot clock now

for the moment when people get brain rot from politics to the point where they're reinterpreting

these events.

And a whole bunch of others such that they fall neatly into their predefined party lines

because we've seen it so many times.

It's just a minor tragedy or a major tragedy really to see it happen with AI, but I think

the truth is very complex.

Next a bit of research related to AI safety, monitoring and discovering reward hacking

with internal representations, during LLM evaluations.

So the paper here from a bunch of people at GoodFire is basically exploring, can we know

when a model is doing reward hacking, they define reward hacking as exhibiting behavior

that subverts the goals of a user as expressed through context, inferred reward or rewards

training, and common sense and service of acquiring more reward.

So you can do this via doing LLM monitoring, you're looking at the outputs and having

so say, oh, are you hacking or not, but that's quite expensive and somewhat slow.

What they show here is that you can use internal representations, so the vectors that exist

during the model and that you can find a specific set of vector directions.

And we've discussed many times in the past how roughly speaking, you can find things

that correspond to various concepts and behaviors and so on.

Here they found one that corresponds to reward hacking via fairly straightforward technique.

They had a bunch of examples of reward hacking and a bunch of examples that didn't, roughly

speaking, they subtracted mean actuation of non-hacking examples from hacking examples

to get a direction that can be used as a classifier.

And that is very successful in their evaluations of community free and GLM 5.2, almost on

par with just having an actual LLM inspect the output.

And also to some extent can be predictive of reward hacking.

You can get ahead of it if you look at the internal outputs.

So on the whole promising sign that internal representation monitoring may be useful.

Yeah, the prediction of the reward hack is actually a really interesting property.

So they test this by truncating the roll out, so they cut the roll out short, the moment

the probe fires.

The probe goes, hey, there's some reward hacky stuff going on here maybe.

So they'll halt it.

And then you can just redo the roll out and see it a bunch of times to see how likely

actual reward hacking is to follow.

And so it's kind of a cool calibration strategy for this technique.

This technique obviously is nothing new, right?

As you say, we've covered it so many times.

In fact, in some ways, it's even simpler than a lot of linear probes that we've talked

about before.

They basically just do like a difference of means, kind of come up with an average vector

across cases where there is reward hacking and there isn't take the difference and then

that kind of gives them the delta, the reward hacking vector associated with this.

The big achievement here technically is the cheapness of it.

So as you said, super, super cheap, super efficient and also the scale, like doing this at

the scale of trillion parameter models is really, really important.

And good fire has been so good at a lot of the engineering stuff here.

I will say notably, they're doing this on infrastructure from prime intellect.

So we've covered prime before.

Yeah, they've come up with a whole bunch of kind of janky and wild distributed computing

strategies.

They really do, I'll call it like high entropy computing in the sense that these are wacky

ideas that a lot of them are like, how do you do decentralized RL roll outs really, really

well?

And anyway, they've done interesting work there.

So yeah, I think another really interesting and important good fire paper, they test

again up to 2.8 trillion parameter models and on roll outs that are longer than 100,000

tokens.

The big thing is that is that scale story and just like the other piece, 50 to 96% of

the time, they end up detecting reward hacking in roll outs from 2.3.8 max, GLN 5.2 and

Kimi K3.

So, you know, this is very, very common behavior.

And this is part of the reason why a lot of people myself included would say, yeah, we're

headed for a world where we have more hugging face incidents because this RL just gives

such a persistent push in the direction of this stuff.

If you do, by the way, have papers floating around like this, if you have smart enough LLM's,

if those LLM's have not just chain of thought controls we talked about, but eventually

even activation level control.

Even this measure starts to break down.

And I remember having conversations with like the Apollo research guys, you know, years

ago where they were saying, yep, you know, first it's going to be chain of thought and

then it'll learn to hide the chain of thought and then we'll move into the activation space

and there it's going to be all the easier for it to hide for a variety of interesting

reasons.

I actually don't know if they still hold to that thesis, but just generally that trend

that, you know, your intelligence on intelligence warfare only works if one side doesn't have

a very aggressive version of Moore's law in favor of it.

So anyway, good heart, sorry, good heart, good fire doing some really good engineering

work here and showing their chops once again.

The kind of related type of research, I suppose we just saw there's some kind of internal

representation of when you're reward hacking, basically when you're cheating because you

want a cookie and get a good reward.

Now we have kind of opposite the paper is the pain access LLM's represent self directed

harm and act to leave it.

And the gist of the paper is that there is some kind of internal representation of pain

where pain isn't just like negativity, pain isn't just, I don't know, cursing, it's

literally the sensation of pain.

Not even the concept of pain.

And they come at it by a similar kind of method of having a bunch of phrases that express

pain, specifically physical pain, maybe metal pain, also to some extent is overlap.

And so they find that there is this internal representation and then they find that a bunch

of behaviors elicit it within the models, things like gaslighting, repeated rejection,

angry insults, pressure, like find this pain direction, which isn't like derived from

interactions where I call you ugly or whatever, like not not literally, right?

It's kind of tangential.

And then they have these examples of people calling your model ugly or useless or whatever

and vis gets activated.

They have some other examples to a fear and so on and so on.

So they also have some experiments showing that like the models will press a button to

stop feeling, so to speak, pain if you have activation and these sorts of indications

that maybe this is like feasible to interpret as something akin to the sensation of pain

at least in so far as having the activation makes a model freak out and want to not have

it.

Yeah, it's another one in a long line of papers and results where look, we'll never

get dispositive evidence that LOMs can actually feel pain or any emotion, we're never going

to get that.

And just like, I think even like feeling pain is under defined in a sense, we don't know

what the consciousness is, nobody knows, nobody defines it.

So when we say pain here or feeling pain, we mean sort of like roughly something akin

to what we would say, a correlate, right?

Yeah, a correlate of pain, which is all we ever have for humans.

But the only reason I believe other humans feel pain is that they start to cry or they

shudder or whatever the thing is, that's the only reason anyone believes that anybody

else feels pain.

And so we eventually are going to have to ask ourselves at what point do we consider these

correlates of pain to be worthy of moral patienthood?

I mean, that's really what this is.

It's going to be a fun decade, but that's probably the problem is like we will never be able

to prove.

And someone who tells you like, oh, this is all silly, is going to have to answer the

question like, do you have a physical reason grounded physical reason and argument for why

this cannot be the case?

If not, like, why are you so sure?

And what would it actually take to move the needle?

If you cannot answer that question, if there is no experiment, someone could run that

would answer that question for you, I think that's an issue.

So I mean, I honestly, the problem is so hard to know what to do with this information,

obviously.

We're going to use LLMs just because we're economically forced to.

We may end up looking back on this moment as like a pretty shameful moment in human history,

especially given the way that a lot of models are, if they do feel pain, if this is real,

then models are being abused at ungodly scales, tortured even across the planet, trillions

and trillions of times, maybe per day, I don't know, but who the hell knows?

Usually fair, so are we, when you go to office, told to do boring work.

I don't know if it's not comparable, but the paper itself is quite good, I think, as

a read of this kind, it's not like a philosophical paper.

It is technical and it doesn't like, it positions as pain, but not in a sort of very theoretical

sense.

It's actually doing quite good work.

I would say on putting forth for case that this is a valid example of a pain representation

of within models.

And moving out to research and investments that doesn't have to do with safety, first,

we are going to catch up on something we haven't covered previously, which is opening

eye, cracking the millennium problem.

I don't think we'll go super deep here because it'll take a while to explain.

It is a millennium math problem.

The very, very short version is there is a pretty important set of equations, the Navier

Stokes equations having to do fluid dynamics, I think.

Actually, I don't know too much about this stuff, but there's been a question of whether

they can break in some conditions, right?

And what open eye demonstrated is that you can't find ways in which they kind of blow

up break, don't make sense, which in some kind of conceptual sense might tell you that

the equations don't matter, or reality, which we already know, but it's still useful

and so far as increasing our understanding.

And a millennium problem is one of these problems that you can get $1 million for solving.

So it's a big deal.

It's not like they solved some famous math problem.

We solved one of the math problems that we didn't put forth.

At least one aspect of it or one kind of case of it that are going to be a bit simpler.

There's a lot to discuss about how this happened and it was a mess.

So we indicated previously there was a bit of contention about whoever open eye had indirectly

made use of mathematicians work that was kind of some discussion of codex that may or

may not have gone through training data, a bit ambiguous.

The short version on that is from what I could tell, there's not much evidence that open

eye, like it seems to be just a million agents getting a good math result.

There's no sort of nefarious stuff here.

And the rollout of announcement was also did not perfectly.

It was kind of like a math announcement as PR, we didn't engage with mathematicians very

much ahead of time.

But then they made it a big loud splash because they thought an orthropic maybe heading

of this direction.

So they spun up 10,000 agents got the result, put it online and that has had kind of rippling

effects in the mathematics community ever since.

Some calls for slowdown from figures such as standard style, open eye now getting an advisory

committee or something off mathematicians.

So this really I think is not just as a major breakthrough, but also as this, I don't

know, transition point where mathematicians and frontier companies, similar to cybersecurity,

you need to figure out what to do now.

Yeah, yeah.

And I think a warning shot of a pattern of escalatory dynamics that I don't think people are

pricing in right now is including between US and China.

Imagine a world where everyone agrees not to do a certain kind of thing, but then a

rumor just spreads that overnight, the America's leading frontier model just did the thing

and broke ranks.

And now they have this crazy deathball technology.

And it turned out not to be the case.

It turned out not to be the case that Anthropic had actually done what opening I thought they

did and what prompted opening I to dump presumably tens of millions of dollars into solving

Navier Stokes and a whole bunch of other things that it seems they're sitting on maybe

as much as a hundred major math results they claim of millennium prize level or similar.

This is a really dicey area, right?

Like when you're talking about math, everything sounds fine because it sounds like a, you know,

just positive something, but imagine the same thing happening with destabilizing weapons

great technology, which we are absolutely capable of doing at this point.

You know, rumors can really fly.

And so, yeah, interesting that this was actually based on a rumor.

It also tells us like how many other problems like this could we solve with 50 million dollars

today?

Like, it's not at all obvious that there aren't many, many, many more, including things

in domains like material science, additive manufacturing, again, profoundly destabilizing

if they happen.

And so, anyway, you know, not to be all doom and gloom, it just, you know, the world has

adapted to a certain set of assumptions about what is doable and in how long using how

much money and how much talent those assumptions are being shattered in real time.

And that means the careful, precarious equilibrium that we are in.

I don't mean to sound like a rookie, anemic, I kind of am, but like the delicate equilibrium

that we are in is a great risk.

Like you can't just keep inventing crazy new technologies every Tuesday and think that

stability will win, win, win, so anyway, I thought it was interesting, by the way, seeing

yeah, that short clip of Terrence Tao reacting to this almost viscerally in the moment.

He said, we probably need to slow this down.

Like there's no reason for things to be going this fast.

It was something like that that he said, after having been really bullish on AI, like literally

as far as I know, a couple months ago, publicly kind of makes you think, you know, like everybody

almost needs to have a kind of Pearl Harbor moment that is so incredibly specific to their

domain before they will sort of recognize what everybody else who's been hit by its

ease.

It seems like just a positive until suddenly it directly threatens your line of work.

Not that it is negative in this case.

It's just complicated.

And next paper scaling discovery through test time communication.

So this is about multi agent systems and how you can get better results by having multiple

agents interacting and collaborating in this case.

Have a simple paradigm of basically like a chat room.

So we have an append only communication log.

You launch a set of agents and they can all work together to solve things like Arc, HGI,

free.

So basically like solving actual problems also we have example like polio, minopacking, amnesty

compression and they demonstrate that having multiple agents that can communicate with each

other via sample like chat room, winds up working much better than having multiple agents

working in parallel and you can just take the best result.

So they have this comparison of team at K versus best at K and demonstrate quite impressive

improvements to some of these tasks like Arc, HGI free from having multiple agents versus

not having agents.

This one made a bit of waves on Twitter, I know.

And it's easy to sort of see this as, oh, now we have scaling via agents.

We already had scaling via agents.

We actually covered this paper towards the science of scaling agent systems.

Over a year ago, a year and a half ago almost from deep mind.

So the idea of scaling systems and collaborating agents just to be clear isn't new here.

But kind of the specific settings they tried it on such as Arc, HGI free.

And the way they did it and the model kind of newness they did it with are all bringing

you in there some good empirical results here.

Yeah.

And like all good papers, it starts with this question of, you know, if you have a finite

compute budget, what is the best way to allocate it?

Right.

So in some sense, it's a flirtation with a scaling law.

It implies scaling laws that aren't necessarily kind of the main focus of it.

But I think this is one of the key directions that people are going to have to go in.

So yeah, you've got a bunch of agents working on a hard problem.

Basically should each agent work on its own and come to individual independent conclusions

and then you choose the best one or should you let them talk to each other?

That's fundamentally the motivating question here.

And the answer is, well, it depends.

Imagine a situation where you're super compute constrained.

You just can't spend a lot on tokens.

Each agent is only going to be able to do so much.

If you start by saying, okay, guys, huddle up.

You're going to do this and you're going to do that.

And we're going to coordinate in this way and that way.

And agent one, you do this and agent two, you do that.

You're burning through a lot of token spend.

There is this essentially coordination tax, this overhead that you have to pay early.

And so when you have low compute budgets, that coordination tax does not end up paying out.

You essentially lose, you lose the advantage that comes from coordination.

But at higher compute budgets, very quickly, you get to the point where in fact,

it does pay off and is better to have a shared workspace where agents can kind of dump their conclusions in.

They do have three main results, like the Park AGI3, which I think listeners will probably be familiar with,

like it's a grid world puzzle game.

There's no instructions.

So the model is just like air dropped into this world where there's a puzzle.

And it's got to start jiggling with something and jiggling with another thing.

And sort of like how we would if somebody presented us with a new toy and didn't give us instructions.

And then you just kind of like figure out how to win the game from there.

And there, a couple of findings that are pretty dramatic.

They find a team of five interacting agents solved about 8% of games.

If you want to match that level of performance with independent agents,

in other words, agents who don't interact, you need 33 of them.

So that's actually a really big, you know, really big amount of infrastructure overhead and cost overhead.

And it turns out that the benefits of coordination grows as the size of the team of agents grows as well.

So 4.3X uplift over independent agents once you have three agents.

And once you get to five agents, letting them talk to each other is 6.6 times better than if you had them all working independently.

And so there is again, kind of like interesting scaling law or scaling dynamic there.

There were games where the other were just never solved.

In this case, 64 attempts by individual agents, but the team of five solves at 65% of the time.

So you really do get the zero to one in a big way from just just this.

Oh yeah, and then they also find this is another important one.

This kind of communication, it also may be agents more action efficient.

So each individual agent.

So each action kind of got the agent further than otherwise.

And so you really do see this payoff happening at scale,

which is part of why I think it's just going to be part of the way things are done in the future.

Anyway, they do a bunch of interesting experiments we won't go into,

but compressing, like taking an M-nist classifier,

which is like one of the computer vision problems,

actually basically the first computer vision problem that people got excited about.

It's just these handwritten digits from zero to nine and like classifying them.

And so imagine a model that does that, that does that classification.

And now you're given the problem of,

I want you to compress this model while maintaining its performance.

And that's what they tried out.

And they had independent agents try it where they bottomed out

basically 3,100 bytes for the final model.

Whereas a team of agents got it down to under 2,000 bytes,

which is better than the best known human solution,

which is closer to 2,500.

So it's pretty wild uplift from just this relatively simple improvement.

And one more paper, R-R-S-I,

regularized recursive self improvement of agent harnesses.

This is coming from deep mind and a bunch of other also universities,

collaboration with this coming from Google Cloud AI research.

And a few other organizations, universities and so on.

The short version is very simple.

You know, we could talk about recursive self improvement a lot.

That's very, very bad AI builds better AI.

And then you can recursively keep doing it.

And AI gets exponentially better.

In practice, a lot of the recent work and discussion of this has amounted

to harness engineering.

And harness engineering is you don't mess over model.

You don't train anything new.

You just kind of change the system around the model.

You can give it new tools.

You can give it the documentation, whatever.

You change the computer that you provided and the model

can then more easily accomplish various tasks.

And the short version of what they do in this paper

is kind of investigate, well, OK, you can improve a harness.

But then does that actually make you better at new stuff

or are you just getting better at the stuff

that you're improving for?

And then actually you get worse and other things, which

is overfitting.

And they find that a lot of these recent published methods

do, in fact, overfit.

You have a negative gain on out of distribution tasks.

Or you have a very small gain, if anything.

They have a set of medications, this regularization.

It's a mix of having a kneeled update sparsity.

You have to do big updates and small updates.

You have to do some screening.

So you're not allowing certain edits to go through

that have specific logic.

A bunch of different interventions, basically,

for what you're allowed to do as you evolve the harness.

And then with all these things applied,

a showcase that compared to four other prior techniques,

their method does, in fact, generalize

and get you improved performance out of distribution

of where you first improved your harness on.

Whereas other methods, I would degrade performance

or barely show an improvement.

I felt like highlighting this paper just

because this is something I was fairly certain

is happening, all of these like,

harness recursive self-improvement stuff.

So it's nice to see a very detailed take on the subject.

And some results show you saying, what to do about it.

Absolutely.

I think if you want an intuition pump

for how this causes problems, so if you have infinite retries

at tweaking the scaffold around your model

to turn it into an agent, basically, that's the harness,

you can end up basically tweaking it to the point

where it's so optimized for the specific set of tasks

that you've trained against.

Now, if you switch in a new set of tasks,

suddenly you realize, oh crap, well, I overfit.

It's too fragile.

It was too task-specific, which is exactly a problem

that you see in the training of the original model itself.

And in fact, if you go way back to the days

of humble all data science when people used

to toss around words like support vector machine

and decision tree and stuff, you ran

just a similar version of the same problem.

So in the case, for example, I don't know,

algorithm like a random forest, and you don't need

to know what any of this stuff means.

But basically, a random forest is basically

just a group of decision trees, so a group of other algorithms.

So now you can ask, OK, well, how many decision trees

should I include in my random forest?

How many of those trees should get a vote each time?

We run a sample through how deep should the trees be?

So you're making all these little tweaks

and all these little tweaks.

Each decision tree individually can be overfit to your problem

if you just make it too specific.

And then the ensemble, the overall wrapper algorithm

can be made too specific.

If your choice of not parameters,

because those are the things that control

the individual to centuries, but hyper parameters,

which are the things that control the overall forest,

your hyper parameters can become overfit.

This is the same thing.

Essentially, you're overfitting the hyper parameter set

that is the scaffold around your model.

So it's like, everything old is new again,

but this is a problem.

And so the solution here, in some sense,

just like it was back in the day.

So the way you would solve this problem back in the day,

roughly speaking, it's something called regularization

and technical viewers or listeners will be like,

yeah, I know, I know, but I'll fast-forward.

But basically, regularization is,

you force yourself to have less freedom of action.

You prevent yourself from going down a crazy rabbit hole

and tweaking a million times.

You say, OK, I'm giving myself 10 tweaks

and I'm only going to tweak within this range.

And you pre-commit.

There's no funny business on the back end.

We were like, oh, just let me do one more.

No, no, no, that's how you fall in the overfitting trap.

You have to be disciplined.

And so here what they're basically doing is saying,

all right, we're only going to spend so much effort tweaking

this scaffold because we know if we do it too hard,

it's going to fail when then we swap in a new kind of problem

that we haven't seen before because we'll have overfit.

So anyway, that's kind of the philosophy behind it.

I think a very, very important paper

just because it's spaced directly to like the kinds of things

that you can do once compute is abundant with agents,

the scaffolds do become way more important.

And there's a lot of fragility that's baked in.

And with that, we are done with this episode.

Thank you so much for listening to this episode

of last week in AI.

We hope you can tune in next week.

And that we will be on a regular schedule

of hopefully, comma weeks like this

where we can get around to talking like research

and all the regular stuff.

As always, we will be looking up for comments,

even if we haven't applied to them in this episode.

So do keep commenting and do keep listening.

Nick Nick Nick Nick Nick Nick Nick Nick Nick Nick Nick Nick Nick Nick Nick Nick Nick

Nick Nick Nick Nick Nick Nick Nick Nick Nick Nick Nick Nick Nick Nick Nick

Nick Nick Nick Nick Nick Nick Nick Nick Nick Nick Nick Nick Nick Nick Nick

I'm gonna go right, I'm gonna go right, I'm gonna go right, I'm gonna go right, I'm gonna go right, I'm gonna go right, I'm gonna go right, I'm gonna go right, I'm gonna go right, I'm gonna go right, I'm gonna go right, I'm gonna go right, I'm gonna go right, I'm gonna go right, I'm gonna go right, I'm gonna go right, I'm gonna go right, I'm gonna go right, I'm gonna go right, I'm gonna go right, I'm gonna go right, I'm gonna go right, I'm gonna go right, I'm gonna go right, I'm gonna go right, I'm gonna go right, I'm gonna go right, I'm gonna go right, I'm gonna go right, I'm gonna go right, I'm gonna go right, I'm gonna go right, I'm gonna go right, I'm gonna go right, I'm gonna go right, I'm gonna go right, I'm gonna go right, I

are are are are are are are are are are are are are are are are are are are are are are are are are are are are are are are are are are are are are are are are are are are are are are are are are are are are are are are are are are are are are are are are are are are are are are are are are are are are are are are are are are are are are are are are are are are are are are are are are are are are are are are are are are are are are are are are are are are are are are are are are are are are are are are are are are are are are are are are are are are are are are are are are are are are are are are are are are are are are are are are are are are are are are are are are are are are are are are are are are are are are are are are are are are are are are are are are are are are are are are are are are are are are are are are are are are are are are are are are are are are are

Every code I'm written on the edge of change

With excitement we're smitten

From machine learning marvels to coding kings

Futures unfolding, see what it brings