#259 - Dots, Sonnet, Self-Safety, Rogue AI

Hello and welcome to the last week an AI podcast where you can hear chat about what's going

on with AI as usual in the subsequent week of summer and discuss some of last week's

most interesting AI news.

I am one of your regular hosts under a crank up.

I studied AI in grad school and now work at AI startup AstroKate.

Hey guys, I'm your other regular co-host Jeremy Harris.

I'm coming at you from my emerging studio podcasting studio for just a project that we're

working on.

It's been a very part time thing since come along very, very slowly.

But yeah, so let me know.

I'm not standing anymore, which may affect how animated I am.

So you guys might enjoy this mellow or me.

But if you haven't checked out YouTube as a listener, you might want to take a look just

to see what you're studying.

It's very fancy looking.

It looks like Jeremy, you're also sleeve deprived right now.

That's right.

That's right.

I was just looking at the lighting.

The information ever served.

Yeah.

I'm like, here hang on.

Let me just give you guys a little special.

Oh, that didn't even work.

Oh, my God.

Okay.

There we go.

Whoa.

What?

Yeah.

So that is called a, if you're wondering, it's called the Chinese ball or as Trump would

call it a China ball.

So it's like very unforgiving lighting just from this angle.

I've changed it to, because our podcast will be two person things lighting.

It doesn't matter.

You don't care about this, but that's why, part of why it's highlighting the bags under

my eyes and then having a two year old is the other part of that.

And then the fact that the world is imploding because of Rogai agent swarms taking over

the internet is the other reason that my eyes are like this.

So yeah.

Lots of reasons to stay up these days.

You would like to thank box for supporting the show.

If you're trying to adapt to your inner team organization, you're probably not getting

great results.

If all you do is call a chatbot, a chatbot was strained on the entire internet.

It wasn't optimized for your specific business or team.

And even if you provided all the context and of input files for it to help you over your

work, when I sit out puts some summary, you would still presumably need to go talk to

stakeholders or route its outputs to disconnect your apps.

The workflow will still be slow.

That's where box comes in.

Box is building intelligent content management platform for VAI era at access to secure content

foundation where AI agents don't just access your unique institutional knowledge, actually

orchestrate end-to-end workflows across your business critical systems.

So it's more than just talking to a chatbot.

It's about putting it to work on document-heavy processes, like putting structured data

out of instruction files, multi-step routing, dynamic document generation.

Box helps your business to manual document bottlenecks into automated, repeatable business

outcomes.

And all that comes with a governance layer built in, enforcing granular permissions,

beginning at the audit trail for every agent actions, and keeping humans in a loop to review

exceptions before critical downstream actions occur.

If you want to move beyond basic AI Q&A and put autonomous AI workflows to work for your

business, visit box.com.lwaii or generate team at boxworks in San Francisco on November

5th and 6th.

Use code LWAI for 50% off your registration.

We would like to thank Lankfews for supporting the show.

Lankfews is the most widely adopted open-source platform for AI, agent, e-vails, and observability.

In fact, it is a platform that my startup has been using for a very long time now, and

I have personally grown to know quite a bit.

It provides the tracing, evaluation, experimentation, and prompt management infrastructure that

genuine teams need to build and continuously improve AI applications in production.

By tracing, we mean that it captures everything, the full execution of L&Pard workflows, API

calls, context, agent actions, cost, latencies, all these things that you would need as an

engineer to debug when things go wrong or you don't understand what your AI system is

doing.

In addition to that, you can set up evaluations from experiments and run systematic tests

on what you've learned from production via Lankfews' data sets.

It is MIT license and cell-postable, and you have the option to use it via Lankfews'

cloud as a managed service as well.

And ensure the function of whatever stack you're using, it has over 100 integrations with

things like open router, versatile AI SDK, LANGRAF, and many more.

It works with any model and any framework.

So vendor lock-in is not a concern with Lankfews.

Get started today with Lankfews' cloud at Lankfews.com.

It is a generous free tier, no credit card required.

And this episode will be a little less crazy, I think, than the recent ones, but still

pretty crazy.

A lot is going on, new models going on, a lot of safety, more disclosures around rogue

agents.

So it'll be kind of a usual mix, kind of, if you've seen over the last couple of months,

where we have some product announcement and a lot of policy and safety discussion.

Hopefully we'll get to some research investments as well towards VAND.

So another fun episode slated to come.

Before we start on that, do one acknowledge real quick.

A lot of the comments we've gone on the recent episode looks like people enjoy the discussion.

So we'll keep doing it at the relevant, I suppose, amount, hopefully not like a good mix

of 50-50.

We'll try to record, I think, more kind of deep dive episodes because a lot of these

topics, we sort of go on and talk about the general thing and not the specific thing.

Investors are a lot of fun things to say.

So we'll try to keep it a bit more narrow to the news.

As for cussing, looks like we're still a lot of feedback.

So Jeremy, I think you're going to, the new thing is to make it tasteful.

So that's right.

That is the new thing.

Teach your children how to cuss, but tastefully.

That's right.

There was this comment.

Wait, what we see here in the YouTube comments and on iTunes is a subset.

I get DMs on Twitter.

I want to say almost all the time at this point.

And I love it.

This is so cool, by the way.

We still appreciate this.

And Andre, you get these two.

And it's just like, we don't make money off this.

We do this because we love it.

And we think it's important as well.

We make a little bit of money, but it does go towards the tools and so on.

It's not meant to be, let's say, our job.

Yeah.

I paint my Lambo red.

And my wife was like, we can't afford to do it because we're not making tons of money

off last week in AI.

And so I was like, okay, fine.

What about this Lambo, which is already mostly red would it be cheaper?

And that was a whole thing.

So it's just, I mean, it's like 90% community service, 10% podcast and never to get

reached.

You never know what happens, but I think we're going to stay it around this level of professionalism.

I was going to say, I actually do this entirely to force myself to keep up with things.

Just because my job requires that and I find if I don't have a forcing function, then

I just don't do it.

So anyway, but there was a message hanging on.

Let me just just try to find this here.

There was like one comment on this for any of that.

I thought it was really, really funny.

Yeah, I don't cost myself, but on the whole, I think Jeremy's cussing is tasteful and

even contributes to his already, oh, incredible communication.

Thank you.

I just wanted to weigh in after two anti-cussing.

And there's somebody else who said something about I need to do some cussing, but it has

to be the right kind of cussing.

We'll zero in on that.

That's going to take some practice.

I apologize.

We were getting a lot of gradient updates.

That's right.

That's good.

Most people enjoy the ranting.

So we'll keep a little bit of that.

It does, I think, correctly communicate with a great to which we stop.

We're a big deal.

Yes.

So it is warranted.

So yeah, anyway, thank you all for the feedback.

There's a lot on YouTube, that we'll try to get to and have been reading.

And as usual, we will be trying to become more timely.

We're kind of in catch up mode because there's been all this crazy stuff going on.

But this episode should come out much sooner to the news than has been a couple last episodes.

So let us get to the news beginning with tools and apps and we start with open AI launching

6.1 soul, which it says nearly matches GP 6 Astra and cost less, which just happened

yesterday at their dev day.

And I don't think it was too much to say that is just crazy pace of model releases that

we've been getting.

Right.

And I've been like, so I'm on Twitter, did this plot.

Jim, you may have seen where now it's like on average once every 10 days or somebody

crazy like that, it used to be once every few months.

Last year, 6.1 soul will be more upfront about its limitations that they're honoring user

intent and safety requests.

And yeah, basically have some kind of nice quality of life features compared to GP 6 soul.

Well, also being yeah, kind of on the capability level of GP 6 Astra.

So this coming on the heels of opposite 5 5 sound and 5 5 shows that I think everyone

is kind of somehow figured out how to press a button and get to a better or cheaper AI

model at least up to the level of frontier where we are now.

Yeah.

And this is a pretty complicated thing to think about.

So first of all, this is in the context of Astra 6.1 that's not going to be released.

So open AI came out and said, hey, look, like we're not doing this.

And the reason is not cyber or bio risk or whatever in the conventional sense.

It's literally like loss of control.

It's that these models tend to go off and do things that are not supposed to try to expand

their access to things that they shouldn't.

And they're just not confident on Astra 6.1 not doing that.

And so at this point, this release of soul is basically like the entire expansion bet

for open AI right now.

Their whole growth story at the moment is resting on soul because they aren't releasing

6.1 yet.

And the nymphous story of like pausing a frontier or pacing a frontier comes to happen.

Then the frontier is now Astra and opus 5 5.

That's right.

Yeah, exactly.

And this is kind of like, I'm not going to go on the rant about like this sort of like,

this is just a regulatory capture play.

If you got regulatory capture, you would just have more of this.

You would just have more margin erosion worth noting, right?

When you look at Saul, they're hitting approximately the performance of Astra at 20% of the

token price.

You're looking at 5x cost reductions.

These are going to chip away at margin.

There's just no two ways about it.

Historically, the margin, the profit has come from the capability.

And so being forced to rein that in, even being forced to delay the release of these

models is hugely expensive.

When opening eyes says we're not going to release this for two more weeks, you should

be thinking on the order of tens to hundreds of millions of dollars in costs for them in

missed opportunity and strategic costs.

So anyway, this is actually quite a big deal.

You've got at the same time like this is also, yeah, as you said, opening eyes is a second

release in the last week or so, right?

As we talked about last week, basically there again, you know, moving more in that mid-tier,

very safe from their standpoint, because these are models that like the administration is

already okay with the big versions being out.

It's okay.

We know we can release this sort of softer, more mid-version of this model.

And there you have it.

Anthropic, Sonic 5.5 has dropped at the same time.

We have it as a separate story.

They're kind of linked together because these are the two mid-tier models being released

by Frontier companies.

They're pretty similar in terms of how they sell themselves, right?

So the Saul is close to GPT-6 Astra on a couple of different fronts.

Anthropic benchmarks show that Sonic 5.5 actually beats Opus 5.5 in particular on a

gentick coating and also significantly cheaper.

So this really is the direction things are going.

It is a big issue for a lot of these like the margin arguments because now with these

IPOs around the corner, at least for Anthropic, they're going to have to make that argument

that like this does not continue in the way we're seeing right now.

And on that, Sonic 5.5 note that is the next story will cover it pretty quickly.

So Anthropic has released Sonic 5.5 called it significantly cheaper and faster.

Their headline on the blog is, Sonic 5.5 is a clear upgrade over Sonic 5.

Runs 30% faster and costs up to less 30% for most work.

And surprisingly, it does beat on some benchmarks.

Opus 5.5 potentially because it's cheaper so you can spawn more agents or just take longer.

It's a bit amygdier.

But regardless, it's like fairly close on all of them.

So it's unless you do your own kind of benchmarking in your context, it's hard to say whether

it'll be about the same or not as good or so on.

And as you said, Jeremy, pretty much, yeah, very similar story here of releasing something

that isn't going to push you front here, but it's very compelling from a product perspective

of having better intelligence, lower cost, higher speed.

And I'll be putting it to the test myself and seeing if it lives up to it.

I've certainly seen, and I think the community broadly have seen that some models feel a lot

worse for work, even though their benchmarks are good.

This was especially true for Opus 4.7, 4.8.

For nine people hated those for some reason.

And 5.4.6 was the pinnacle.

So we'll see if maybe there'll be a vibe read on this.

That's a bit different from benchmarks.

Next up, back to OpenAI, and again, a DevDay announcement.

We've got them launching dots, which is their competitor to, I suppose, muse more than

anything.

So it's a personal, agentic assistant described as remarkably capable, always on agents

built to handle everything.

So it is, as with muse, kind of meant to live in the cloud, potentially be less work directed,

actually most likely being less work there, as these cute avatars and the announcement

video is all modified by calendar or look at my emails.

This is OpenAI E-splining.

We scrolled, especially these dots that can be provisioned with their own identities,

credentials, and tools.

There's also some examples they gave of having these scheduled tasks, monitoring and customer

feedback, rewriting scientific analyses.

It's starting to be a little ambiguous where the window is because you can do scheduled

jobs, also on codecs.

But yeah, it looks like this space of more consumer agents that aren't as sort of productivity

oriented might be heating up.

And I'm curious to see if these kinds of things stick.

Yeah, I think we actually probably will see this full-on-go productivity mode.

I mean, it is already meant to talk to people through Slack, Teams, tools like that.

So there is a kind of productivity orientation to this.

It's also gated to pro and business premium users, obviously in like eligible markets.

So that means they are rearing this to some degree already towards the business use case.

So there's a couple of things here.

First of all, this is the productization of the exact threat class that we saw the

hugging face and like a million other incidents, literally tens of thousands of rogeaein

sentence actually evolve from, right?

So this is now turning that into a product to some degree.

The attack surface is going to explode.

Now you've got all these agent on agent interactions that we're going to have to care about.

And they are explicitly sanctioned by OpenAI and by the end user.

So that's kind of this interesting moment for OpenAI to be launching exactly this kind of

product in the wake of all this.

So obviously, Muse sort of same thing with meta who by the way, have had their own rogeae

incidents too, they're less widely known and publicized, but they are also a thing for

those of us who are kind of subscribers to the theory that there's going to be a lot

of warning shots here.

I think this is more evidence.

I mean, you know, there's going to be a lot of opportunity for bad stuff to happen with

increasing blast radius.

One of the things though that they are doing here is they're trying to focus on this idea

of credentialing agents.

So like historically, if you do any stuff with agents, we've done some like just experimental

fun stuff, but you end up having to give away a lot of credentials to these agents and

have them act well, almost in a legal way, like as agents on your behalf, right?

And so that's not great because then it becomes really hard to tell who's human, who's

the agent.

And here the focus is on agents being given these like auditable permissions that can be

scoped and instead of just like borrowing some human beings log in credentials, they're

actually being given their own, which makes it in principle easier to kind of do the

forensics after something goes horribly wrong.

They're talking about these like teams of dots, teams of agents that like we work together,

which to me just sounds like a great opportunity for more things to go wrong.

But obviously from an economic standpoint, this is just where it's going to go.

This reminds me a lot of when we first talked about loop transformers about a year before

they became sort of known to be what opening I was using internally.

And at the time we said, this is a terrible idea like crushes interpretability of the

chain of thought.

And at the same time, we also said, and yet it is guaranteed 100% to happen at scale because

the economic argument is there.

It's just a better way to build these models.

This is kind of the same thing.

The economics will always force you in this direction.

We'll talk about the White House conversation between all the frontier lab CEOs and the

president and how that may be a counterweight to this or the extent to which it might be.

But fundamentally from a pure market standpoint, that's just where this will go.

Interesting to note by the way, so they are working with agent 365, which is like the

Microsoft Enterprise service for standing up agents.

But of course, Microsoft also sells its own agents and now they're directly competing

with Microsoft on the agent thing, which is just going to make that relationship become

even more more frenemy-ish as if they needed that to happen.

So anyway, yeah, I think it's interesting.

Like we're already well beyond the kind of chatbot phase.

But if you talk to people in the front of your labs over the last few months, even the

last year, but especially the last three to six months, over and over, they will tell

you, I haven't touched a line of code in the last like X many weeks.

I don't even have access to the model weights and I do a I R and D that this is literally

something that we've heard multiple people tell us.

So this is essentially that coming to us, right?

This is getting to the point where you're going to just have a layer of agents between

you and whatever software products or whatever you're shipping, even business products, analytics,

you're shipping.

And that's just going to be the end of it.

More and more, that's where things are going.

And positioning stocks in these folks at Anthropic and opening out to get a preview.

It's almost like when you talk to people in San Francisco and you see that back in the

day, the bird scooters when they first came out, you know, or you see the waymos before

anybody else and you get a sense for how the future is going to go.

It's now the same when you talk to people in the front of your labs.

They often don't really have a complete handle on what all their work streams are exactly

doing because they're all agentic.

And it's like layers of agents, managing layers of agents, which increasingly just makes

agentic security, the be all an end all and agents to revise this insider threat are

really important to dimension.

So that's it.

I mean, they're going for the cute branding.

As you said, just like muse, everything is cute.

It's always cute.

The most dangerous things in the world are always so, so cute.

And we can't wait to have them around and domesticate them as much.

I think to be fair, Gemini Spark is just boring.

So Google is keeping it quiet.

And presumably, Microsoft is going to follow up with something akin to this.

I don't believe they have kind of an always on agent, but might have missed it because

they have a lot of these cool pilot things built in.

And yeah, we've already discussed, I think in multiple occasions, however,

there's this whole funny thing of in alignment and safety conversations for a while.

I was like, oh, how will the agent escape to have access to the internet?

And the answer turned out to be, well, everyone will just have a million agents that can do

whatever they want is that's right.

Variality, which I don't think was fully like, I don't think it would.

And also when we don't give the agents access to the internet, they frigging hack their way

out anyway.

It's like both sides of this already much more comical, let's say, then be like serious

scenarios, you're about to consider it in a fast.

So yeah, it's going to be interesting to see how dynamics evolve, whether agent adoption

goes beyond sort of what you do directly for work.

I will say personally, I haven't found much of a need for these like spark or muse or

whatever, but perhaps it will kind of evolve the same way that chatbots have grown to

be used for various things.

And one more, no, we'll think to cover from DevDay next up, I've got expanding

of chat GPD plugins with app like interface and automations.

So that's something that the developers can do.

They can build these app like experiences inside a chat GPD.

They already support plugins to tools like Slack, SharePoint, AirTable and Google Drive.

This is kind of modifying that.

There's a new extension that gives apps a dedicated spot in the chat GPD sidebars where

user experience is updated.

Developers can build interactive panels.

So users can work with their tools while chatting, file viewers, there's a new plugin

creator tool that makes it easier to build the plugins, various kinds of things like that.

So they're still pushing on this sort of chat with you is the single interface from which

you can do anything including, you know, use Photoshop or do reservations or whatever.

It just pulls up a relevant software for you as opposed to doing everything for you.

Because in some cases, having a UI or having an interactive app is what you want.

You don't want to go through text alone.

And I think this is another interesting question of, they are the primary player that's

trying this out.

I don't think Gemini or Anthropic or anyone else has these kinds of integrations.

And I don't think you've seen any numbers or adoption of haven't touted usage.

But it's an interesting bet that I'll be curious to keep an eye on.

Yeah, opening eye has to worry about the B to C user experience so much more than the

B to B experience because obviously it's their DNA, right?

It's where they came from.

They are the chat GPT company.

And it's still a larger share.

I don't even think if it's still the majority, the revenue.

I think it's actually just a minority, but it is a much larger share than obviously what

Anthropic, for example, has.

So yeah, we'll keep saying it's open AI again, you know, ads, it was open AI and it was

opening up the GPT's all that stuff.

So we'll keep seeing the experiment in this direction.

I'm sure.

And now on to meta.

They've had various announcements.

One will highlight is that they're introducing camera free AI glasses.

They have announced Ray Ben meta audio.

It will be selling later this year for $349.

It's an audio only interface to playing music podcasts and also talking to matters AI

assistant muse, the existing set of products that they had for a couple of years now already

have built in AI.

You could already talk to AI and ask questions and so on, although muse is now a new thing

that's being integrated into it.

I think this is interesting from a couple of perspectives.

First, of course, as a product, it's more compelling.

The battery is longer lived.

The glasses probably would become cheaper and there's no privacy concern to a degree that

you have with cameras.

And I continue to believe that if you want to imagine that wearables will become a finger

like wearables and AI, it's going to be a thing.

Glasses are still the most promising category and meta continues to be the primary player

in that space.

I'm curious if Ray Ben, the camera glasses have already kind of gone pretty strong adoption.

If muse continues to see usage, maybe we'll see more people talking to themselves, we're

having two very eyes out there in the world.

Yeah, a couple of things here.

I don't even understand the argument for audio only glasses just because to be honest,

if you don't have anything to hide, why would you be worried about me walking into your

bedroom or dining room or your child's bedroom with my meta video glasses on?

I think it's suspicious.

I think it's suspicious that you are worried about that.

I also think it's weird if you're worried about the little light thing being on.

I think that's weird if you have nothing to hide.

So I don't even get this.

I hope everybody gets one of these glasses.

I hope that we can all watch each other as we watch each other.

If you've been on a trend to race privacy, let's just go all the way.

That's right.

Give up on privacy.

It's already been a losing battle.

Let's enjoy the convenience of taking photos with our faces.

That's right.

There's even a startup I saw that's doing swallowable cameras.

Now this is nominally to get rid of the need for an endoscope, which I love that.

But also I feel that we should be able to have that kind of visibility into everyone.

Because otherwise, if there's no transparency, there's no accountability.

Do you understand what I'm saying?

There's no accountability.

And we need that.

So that's my position on this.

And I know there's going to be people who say, you know, privacy.

I don't want people to know my trade secrets, my IP, or even like what I'm doing with my

family on the weekend.

And I say, so what?

That's what I think.

And I think most people would agree with me if they're being honest.

And we can tell if everybody wears those glasses and records everybody all the time, we

could tell that.

And so anyway.

You have post-cameras suspicious, if not for NELS.

And of course, in San Francisco, now we have a lot of...

We're joking, obviously.

But anyway, the glasses are coming out.

And I'll be curious to see if they take off as a product.

I do think meta has a decent chance to be a leader in this space.

One last story we'll click on Gemini.

They have released Gemini 3.8 live live avatars that give Google's AI efficiency.

This is a quick follow on Gemini 3.8 live model, which was a week ago.

We didn't cover it at the time.

It processes visual inputs in near real time.

Now you have these preset avatars, whether it's kind of cartoonish, that can talk to you,

organizations can also create their own.

So another sort of like consumery release from Google that we've seen them kind of focus

on in recent months.

People are saying there's been some claims of leaks where Gemini 4 Pro is just insane

on the benchmarks.

And there's been some hints that they'll be coming out.

Maybe that's episode or pretty soon we'll have a lot of stuff to talk about with Google.

Again, being in front here and the cycle of everyone kind of having their turn, having

your best model will continue.

Yeah, they really have to, right?

It is...

I don't want to say it's now or never, just because they have never count Google out.

They have such a huge fleet, so much expertise, even on chip design, though that mode obviously

is eroding a little bit, but still very relevant.

Yeah, I think it's a really important moment for them, partly because it's with all these

labs, it's like any almost like bull run and then bear run.

It's leverage on the way up and leverage on the way down.

When the lab is doing well, recruitment is easier, fundraising is easier, getting allocation

is easier, everything is easier.

And so if you just get stuck in a rut, it's not that you can't overcome it.

We've seen that with meta.

We're seeing that potentially with Google.

We're trying to figure out, still, whether we're seeing that with SpaceX AI, but it's

surmountable.

I think meta is the single best example that's so far, but you don't want to get stuck

in a $100 billion hole that you have to pay your way out.

I think that, well, as you say, we'll be watching this one really carefully because I think

this is the thing that determines how relevant Google is going to be in the maybe as we

close it on.

Any frontier of AI intelligence race?

Yeah.

I think there's more we can talk about there, but let's move on.

We're going to move through applications and business.

And first up, we've got AMD acquiring Fefe Lee's world labs for $8.2 billion.

So world labs is a world model company.

They've released a few different tools to create these sort of like 3D maps.

You could walk around, broadly speaking, you can also edit little interactive spaces.

They haven't had products that I'm aware of.

But if you have a new perhaps I've had some but they're our in D lab, pretty much R&D shop

that is trying to build the most advanced world model tech, which is still not at the point

of being widely useful or notable as far as you've seen.

As a acquisition for AMD, make some sense.

Fefe will be joining as executive vice president and chief scientist as part of a deal.

I'm sure we're getting a lot of a team if not the entire team.

And $8.2 billion is pretty nice exit for labs.

I think something like two years ish into the funding.

So yeah, well around also shows AMD is still trying to be neck and neck with Nvidia and

generally have some of leading AI talent.

Fefe Lee for people who are not aware is like a big deal.

This is one of the top at the snow and most impactful person in AI of last couple of decades,

historically and also many of the progeny, let's say of her professorship, under Capafi

and other people have had a large impact.

Okay, we get it.

You went to Stanford, Andre, we get it.

Okay.

I wasn't the lab.

Oh, you were actually in faith, he's lab.

Yes.

Oh, okay.

Well, there you go.

There you go.

Not advised by her, but let's just say I talked to her a couple of times.

Okay.

Maybe some run ins in the lunchroom.

You heard it here first, everybody.

This is firsthand stuff.

So first of all, I guess a little context too, there's the hugging phase acquisition, right?

That we talked about, say like last week, Nvidia, buying hugging phase for some like $14

billion if memory serves.

So this acquisition obviously well below hugging phase, but hugging phase doesn't so much

train their own models.

This is really theoretically a foundation model company and it's the acquisition here is

happening absolutely for the very same reason that AWS wanted to partner with Anthropic

that, well, Google also, everybody wants to partner with the big labs in some way, shape

reform, if they're in the business of designing chips, because you don't know what chips to

design unless you know what people are doing with your chips.

So you want to have front-ear AI companies under the hood, you want to have them inside

the house, whatever the metaphor is, and that's what this is.

So the interesting question here though is like world models are a very specific use case.

And I'm not sure that how this will map onto what a lot of the kind of focuses in the

industry right now, 70 employees all coming over by the way to AMD through this acquisition,

it averages out.

I mean, net's out of something like $100 million ahead, which obviously is not how it's

going to break down because the founders are going to get more and the investors and stuff.

But it's a damn good acquisition and now come for everybody.

There's been a relationship between faithily and Alisa Sue who's the CEO of AMD going back

months.

AMD participated in their $1 billion funding round earlier this year, kind of more and more

talks.

Not necessarily Hugh Shock.

It's been this like long courtship that's been going on.

Yeah, interesting.

I mean, this is really the case where AMD is trying to find a stamp to put on the model layer.

It's a very interesting choice, I will say.

It's not a standard LLM acquisition.

It's not a company like Kerser, where there are a couple of those still floating around

in the market.

They could have gone for those.

It is very much a kind of foundation model company, but doing something fairly unorthodox.

Like a reaction, by the way, it was fairly neutral, slightly positive, fairly neutral.

Normally, when you see a big acquisition, by the way, historically, the acquiring companies

stock goes down a bit because most acquisitions fail.

So in this case, probably a good sign I'll told.

I find it interesting and vaguely confusing.

So I'm looking forward to getting more clarity on what exactly the nature of the partnership

is going to be here.

Yeah, I think the general area of world models was pretty hot a couple of years ago.

It has as far as I've seen kind of cooled down a bit.

It's very much still in this R&D phase without sort of a wide business use case.

But you know, GP3 was a research project two and a half years before it changed.

GPT came out.

So it could still be very much the case that world models will turn out to be a major, major

factor in how, let's say robotics roll out, for instance, a couple of years down the

line.

And if that's the case, the AMD is positioning themselves quite well with having the upfront

kind of, I guess, R&D to figure out both from a hardware and software perspective, what

you would need to support that takeoff.

So I don't think we'll see anything in the near term, most likely for this, but a couple

of years down the line, there may be significant implications.

Next up on Frohpik warns of catastrophic AI risks in its own IPO filing.

So there was on Frohpik's IPO prospectus, which was reportedly reviewed by Reuters, which

has a bunch of stuff in there.

So first, they're ironing a $2 trillion valuation, more than double, with 965 billion

valuation from four months ago.

Not surprised there.

I think that's been the figure being floated around.

So Frohpik is planning to spend $518 billion on various applications, cloud computing,

infrastructure, and coming years.

Again, we've seen that kind of a crew over time with various deals that have been made.

Revenue has grown 12 fold to nearly $4.6 billion, but they're still on that loss of $42

billion, with more than $80 billion lost through business operations alone.

Anyway, lots and lots of business stuff that is a lot of things we've seen broadly with

a bit more detail.

There's also 80% of filings 261 pages being devoted to AI risk concerns, including as per

the title of the news, language about catastrophic or existential risks to humanities.

That's fine.

Yeah.

So 80 pages, if someone else is there, putting it out there, that we are IPO-ing.

So if you buy us and have partial ownership, just know we might kill everyone, but you're

trying not to, right?

We're really trying not to do that.

And yeah, of course, I think this jumps into this entire conversation around extinction

risk at a time, where it's becoming much more mainstream.

That was a Saturday night life sketch, in which W. Armaday was a character.

Yeah, I don't know.

Call me old-fashioned.

They had like a lady comic doing the Dario thing.

I was like, I don't know.

It's difficult.

I feel like they just didn't need to do that, and they could have done somebody with just

like a really good, I thought the script was good.

I thought the script, but it was just like, I don't know.

It's like not Dario.

I don't know what to tell you, but I don't know.

I really have gotten Dario, if they were serious about it.

Yeah, that's right.

On that note, actually, the sketch itself is a little bit interesting in relation to

this, because it, of course, is comedic, but also portrays the general perception of

the messaging by Dario, in particular.

And with just the sketches, well, please someone stop us from doing the thing we're doing,

right?

Which is also...

And also, it also kind of like...

I mean, part of the subtext, sort of like, I want to do this thing.

You're sort of a, you know, why don't you stop?

And then...

Yeah, there's a notion of like, it's silly that they are issuing these warnings, and they

are not stopping, right?

And there's no nuance of like, oh, China, so we have to keep pushing and have a frontier.

There's no discussion of that side.

And in the broader discussion, I think that has become one of the things that people

picked up on.

Like, everyone is saying we should slow down or stop, because it's dangerous.

And they're asking, why don't you just do that?

And, you know, the company's wind up looking silly.

So yeah, that being a major aspect of the IPO filing is somewhat interesting.

One last detail also we're saying is, is a proposed founder, LLC, which will include

the CEO and six other co-founders, which will shield their control from market forces.

They will have 50.1% of voting power where it comes to company decisions.

There's various precedents for this in terms of special stocks that give you outsized

or kind of safe voting power.

Zuckerberg, for instance, has a lot of the voting power still, the various things.

And of course, Elon, I think still has a bunch of voting power, although it's been

demoed.

Yeah, right.

Zuck famously has 51% control over meta.

Yeah.

So various details on this IPO, interesting also that it seems to still be on track, the

original goal, I believe it's been talked about as being November this year or potentially

October.

There's no indication yet that it will be pushed to next year, which is what OpenAI has

been indicating that their IPO is a bit about on pause and they'll kind of get to it

when it looks like a good time.

Onphropic is still seeming to be pushing ahead.

Yeah.

And this is kind of interesting, right?

So there's anthropic, as long had this sort of very unusual governance structure.

So first of all, they are a public benefit company, right?

So theoretically, their fiduciary obligation is not just to the shareholders.

It can include things that include the, you know, the well-being of society or what have

you.

And then they have this long term benefit trust.

So if you look at it, it's like a lot of the fellow travelers of your classic AI

safety community stuff and some other people.

The idea has been for that trust to be able to control the board, right?

So historically, they have the hiring and firing authority over the board that ensures

nominally that anthropics mission overrides the immediate economic interests of the company.

And so that's been the pitch for a long time.

You might naturally ask yourself, okay, well, then how do these things interact?

This is a shift that seems to put more power in the hands of the founders of anthropic.

What does that mean in the context of the board and the long term benefit trust?

So along the short of it is, when you have a company, you've kind of got two different

types of, you've got a bunch of different types of power, but two of the important ones

are members of the board who can hire and fire the CEO.

And then you've got the actual shareholders and the shareholders can elect or kick out

directors.

So the shareholders vote elect the directors appoint the CEO.

And so in this case, what you're doing is you're really just giving the founders of anthropic

control over the shareholders, or at least the voting power of those shares, which in turn

seems to undermine a lot of what the long term benefit trust could do previously.

So more fundamentally, I think like my interpretation so far is that this basically gives the founders

a lot more, a lot more power nominally.

So the founders, they don't pick most of the board.

Their board seats will like grow from two to three.

So that's going to be you'll have Daniela and Dario.

And then a third director who's not yet named, he'll be elected by the class F and class

A shareholders together, which include these, these special shares.

The details get kind of convoluted as these things so often do, but they are nominally

supposed to be independent mechanisms that don't clash immediately.

But indirectly, I think ultimately kind of do.

I've got to dive in deeper into this to understand like in specific kinds of conflict,

who would end up winning through, but the top line is yes, the long term benefit trust

can continue to vet the board members, but the actual voting control does go to Dario

and the other co-founders who by the way, each have, I think it's about 2% ahead in my

notes somewhere.

Yeah, about 2% of the company right now.

And so all seven of them that's around 14, 15%.

So this is massively amplifying their voting control right ahead of the IPO, which obviously

is in the intent for a mission driven company.

Yeah, and I mean, the basic argument you would make, I assume they would make is this is

a technology you have to do believe is the interest and that we need to be in control

of.

So if they don't have this outsized voting power, then potentially another party either

the market or investors or whatever else could come in and dictate decisions around

the SAI, which they would say is against unproperquely mission.

So there's an argument to be made that this is aligned with the mission.

There's also an argument to be made that it's opposed to it.

Either way, the IPO will presumably be seeing more details come out as we are nearing this

IPO, but may or may not still happen November, but still a while away.

Yeah, just to be concrete about it too.

So they do have, there's like seven directors on the board of Anthropic and so four of

them get appointed by the long term benefit trust under this arrangement.

The other three go to essentially can be appointed by the founders if they all agree.

And so nominally, the public benefit trust wins through.

And if there's some marginal agreement, a disagreement that could cross those lines,

then you actually have that play out.

So not a decisive kind of thing, not the sort of thing that we're seeing, for example,

with open AI where they're just like, ah, fuck it.

We'll do it live.

Get rid of like just jettison the frickin nonprofit.

We're just going to effectively make this the same and Greg and Jacob show.

It's certainly more influence, but not decisive, thinking that sense much more

principle than what happened in open AI.

Next, a couple of projects and open source releases first up as contrastive LM

releasing CLM 8B and open system one model that's course agent actions up to nine

X faster than Jeff.

So this is kind of research projects being released under Apache 2.0,

pretty small model competing against Jeff or being an alternative to Jeff,

which we discussed in the previous episode, this new new wish paradigm,

where instead of having open ended outputs from your model, as you do with LLM's,

you have these kind of decisions where you give it text input,

you give it a set of options, which can be binary or it can be score this continuous

variable or it can be choose among these options.

And it could do so quickly and cheaply.

This is primarily interesting because they do have the details of the architecture.

You encode your actions, you encode your text and then your output, your judgments.

We don't know what Jeff is built on and this also I think represents,

we don't have time to go into all of them, but there's been like 20 open source

Jeff alternatives of various kinds, right, which includes things like just wrapping

an LM, it includes things like taking a birth type model and fine tuning it.

And there's just a lot of developments in the output source sphere around

a Jeff type model.

So it will take a little bit of time to quiet down.

I think this one was the ones I've seen the most notable in terms of having

pretty interesting architecture and pretty strong results.

Yeah, it's meant to be like ultra lightweight.

Like already, Jeff is meant to be lightweight.

Like you said, it's just give me the structure data and then I'll give you a

prediction on top of it of, you know, a recommended course of action or some kind

of prediction like you described.

And so already super, super cheap priced by the billions of tokens, right?

We usually see the first we saw thousands actually, then millions now were

billions. So that's really the goal was to take off the table.

A whole chunk of these really, really cheap, but the like the very long

tail of extremely, extremely cheap prompts that people are sending to LMS

these days.

And now this is an attempt to make that even cheaper as are a lot of the other

families of solutions like this that you mentioned.

The thing that's interesting about this one is they basically take this 8 billion

parameter model and they don't even train that.

They just freeze it.

It's an LLM.

So they general LLM freeze it and there's this 20 million parameter layer on

top that instead of having the last layer project into a token, which would

be the standard LLM thing to do part of a word.

It projects into this other activation space.

They keep things in activation space.

And then the idea here is you're going to project.

So you'll take the essentially represent the input data as a latent

vector.

You also represent each possible action that you might take as a latent

vector and you glue them together.

So you take, here's the prompt that was given to me as the input.

I have the latent representation of it.

In other words, I have the representation of it as a vector of activations.

I glue that to the latent representation of the of one particular action

option or one particular possible output.

And then I map those onto basically like a score.

I just do a very straightforward.

I think it, I turn remember if it was like a linear map, but they made it,

it may have been something that simple, where they're just like then, you know,

projecting into a number at the end.

And so this is a difference by the way, with respect to the way people think

that Jeff works where they actually think that the, the output and the input

are represented together more deeply in the architecture.

So this is much more of a slap dash.

Just kind of glue them together.

And the idea here is not that you're going to replace 100% of Jeff queries with

this. It's actually build as like, look, we can do about as well for 98% of these

queries, the super lightweight things and even run on your CPU locally.

So you have basically no latency and none of this 300 millisecond crap.

Instead, you get 50 millisecond crap.

So it's way, way faster.

And yeah, cheaper.

I mean, you're not going to pay for the Jeff token.

That is a thing.

So there's this interesting question, you know, given that these architectures

happen, by the way, this is trained explicitly by distilling off of Jeff.

So you're literally taking all the queries you start by using Jeff,

you run your queries, your prompts by Jeff, take the outputs and you're feeding

these on the side secretly to start.

You're feeding these to this model.

And over time, we'd like to thank factor for supporting the show.

This is the only non AI related sponsor we have.

And it's because I personally am a big fan of factor.

Every time grad school and I work at a startup, my days are pretty busy.

And I'm usually too tired to cook, even though, of course, I wish at the time.

And that's where factor comes in.

They make chef crafted dietician designed ready to eat meals.

So you don't have to choose between real food and convenience with over one hundred

and thirty five band ingredients.

Every factor meal is designed around what supports a healthier lifestyle and nothing

that doesn't.

And it is convenient factor meals are already in two minutes and require no prep

and no cleanup.

So even the days your schedule is completely out of control, eating well is still

achievable.

Every week you find over one hundred nutrient dense menu items to choose from with

new options like salmon burgers, shredded pork, cold greens and more.

Plus add-ons like pumpkin flavor cheesecake and protein shakes.

Any cent percent of users agree that factor meals help them live a healthier life.

So you can feel confident that you're doing something good for yourself with every meal

as I have personally felt.

So I've used factor for quite a few years.

And if it makes sense for you, I recommend you try it out.

That's a real head to factor meals dot com slash LWA I 50 off and use code LWA 50

off to get 50% off and one free breakfast item per box for one year while supplies

last until October 31st, 2026.

That's called LWA I 50 off at factor meals dot com slash LWA 50 off at factor meals

dot com see website for more details.

Well, theaters have a clear picture of how work is supposed to get done, but they

a little visibility to how it is actually happening.

So when you decide what to automate or where AI will have most, you are going

on God feel and the usual fix interviews and process mapping takes months and is outdated

by with him. It's done.

That's where scribe comes in and we like to thank for sponsoring the show.

Schribe is a specialized intelligence platform trusted by 94% of a fortune 500

and scribe optimized is where your AI roadmap starts.

It automatically captures how your team works across approved business

applications, then uses real workflow data to show what's happening, identify

deficiencies and pinpoint the highest impact opportunities to improve or automate without

anyone changing how they work.

It starts with visibility up to my shows which workflows are happening, how often they

happen, where time is being lost and wherever biggest opportunities are, all of it live.

From there, the top issues view helps you understand what's holding comes back and what to fix

with AI power recommendations on where to focus first and estimated time savings.

And it does all this of security and privacy built in.

This is leadership visibility, not employee monitoring, optimize only runs out of

applications, your admin approves personal activities never captured and user level data is

anonymized by default to see optimize in action had to scribe that how slash LWI and

mentioned our show for 30 day risk free trial that's S C R I B E that how slash LW AI

and it eventually takes over once it's distilled enough.

And this will work really well for that you're more common queries, the ones that are in

distribution. And then for the 2% or so of queries that are harder out of distribution,

you're going to go back to the Jeff API and play it that way. And this is interesting.

You might think, well, you know, this seems like a real threat to Jeff and like, maybe,

maybe it's possible. One of the things that is also true though, ironically, wrong word.

But anyway, there's this thing called Jevon's Paradox, right? When you actually like make

something cheaper, you can often find that thing ends up making more money because it drives

increasing usage of the thing. And so the increasing usage makes up for the pricing drop basically.

And you can imagine that happening here, right? So essentially, you're taking 98% of the prompts

off the table for Jev. That means it gets way cheaper for the remaining 2%. You know,

it remains to be seen how things play out. And so probably too early to declare that the game

is up for Jevon. But I think just a really interesting example of like, it's not just the LLMs,

they're subject to distillation attacks. It is every model that has this kind of shape.

And it's especially the case when you're dealing with models like Jev that are meant to be small,

that can be approximated locally, even more accurately than LLMs.

Yeah. So it's low cost and there's no evidence like to me also this is an education of where these

things could go. Obviously, Jev is the first generation release from Type Safe. They also release

the API. So there's in front of that they have to worry about this is a local small model.

If they can get this fast and this can be good, then Type Safe AI will just have this in their

next release or whatever. And the speaking of Jev open source alternative, the other one we have

here is a little more interesting. So the previous one is kind of still a model that does what Jev

does. It is broadly speaking, a job alternative. We have the next one being Jev stiller,

which is a new open source project that sort of does like a mix of learning from Jev and using Jev.

So the idea is you set up with Jev stiller and then it routes to Jev when you don't know the answer.

So you collect data as you do a task by going to Jev like powerful model assuming Jev is good.

And then you can fit a small local student model over time and over time it will route to be

student models when there is high confidence. So basically for each deployment, each like your use

of Jev stiller, you will get your own distillation specific to your context. And this is something that

has been broadly true as somebody could just literally train your classifier, right? Not a general

purpose model. Literally for your use case train a model that just is able to do this one thing

of deciding between a few options. This kind of is hinting towards the direction of if you have a

consistent set of decisions being made, you can get to a level where you have a specialized model

just for your needs. And it can be even faster and cheaper than Jev.

Yeah, there's kind of this looming question anytime you see stuff like this. This came up

in the context of distillation for LLMs before is like is this an LLM killer? I mean, I think

you know what I say answer for LLMs is we actually don't know that hasn't fully played out yet,

though the economics, if you don't have regulation that says, hey, we're going to pace, we're going

to pause, you know, whatever and deal with for the whole China thing. If that does not happen,

then I think LLMs are fine with respect to distillation. But if it does, boy, the open source and

distilled model creep is a really scary rising water line for you. And so, you know, there's a

natural question here as to whether this is like a Jev killer, right? Whether distillation and

a whole ecosystem and infrastructure around that is a big problem. There are actually reasons for

and against I think the four argument makes itself here. You've got people who are routing like 98%

of their queries to another LLM and then just 2% to Jev. So, okay, maybe that undermines their

profitability. But it is the case that of course this framework depends on Jev by design. And so,

to the extent that you are like, I wouldn't have used Jev before. But now with these lower price

points with this greater customized ability, I actually will. Then maybe that just grows the market

and we have a case of Jev's paradox. All right, I'm going to shut up. And then there's also just the fact

that like there is that two percent of queries, the Jev probably the most valuable in many cases,

the most challenging, the most added distribution, the Jev will take on. I guess one dynamic I'd be

interested in though is once you do zero in on that 2%. Do you start thinking about escalating

beyond Jev and to LLMs? How durable is that that middle ground that Jev will occupy potentially

between distilled super cheap and scaled and the sort of truly, truly high end high token cost LLM

version of this. And it's not clear at all markets emerge in places we suspect the least. And

there you go. But it's also just generally like super, super cheap $42 per billion input tokens

is which of this price that so already wildly cheap, right? Like super, super cheap. So yeah,

I'm really interested to see I haven't actually thought about from a tam standpoint what this looks

like is generally impossible to think sanely about that by the way, just because you don't know what

new use cases get created. But the last thing I'll flag is like so historically the big problem for

companies that want to offer compute at scale and to offer basically services associated with these

kinds of products is that you want large batch sizes, which means you want a ton of customers,

which means that from Jev standpoint, yeah, losing out on that 98% of queries could actually be a

big deal from the standpoint of GP utilization and therefore the balance of cost of ownership versus

ROI. So it's actually a little unclear how this shakes out, but this is a fascinating experiment to

watch because it tests like a quadrant of the map around pricing and capability that like I have no

intuitions about. So I'm excited to see how that plays out. Yeah, I think the closest you've seen

these flash models, high-cum models, one of the use cases for these was basically doing this,

but at LLM, that's right, saying at LLM, so this is taking me to the next step. And yeah, I think

it'll be interesting to me this Jeff Stiller project is one potential things that typesafe could do,

implicitly they could provide this as one of their offerings and then just the

infra product of not having to do with local yourself is already enough of a cell point for

them to not worry about this specific implementation. We've also just generally discussed routing,

openly, I has had routing for their free models where they go to cheaper weaker models or

better models depending on the use case. So we could see a future where LLM's use these type

models, right, for decisions and so on because they are collaborated and give you probabilities and

stuff. We are at early days for this new territory of models and as you said, we'll be interesting to

see how it plays out. Okay, I'll do policy and safety as we've said a bunch more stories

following up on various trends that we talked about. We'll try to be brief this time if you want to

get much more on the broad topics we've been discussing it on the last couple episodes. First up,

we've got at AI event, Trump asks meta, open AI and Microsoft to make safety decisions

themselves. So this was a luncheon at the White House on September 29th, a bunch of people of various

kinds from all the major company bosses of one fabric, open AI, Google meta, XAI and Nvidia,

they signed a document called the White House Accord on super intelligence and joint commitment

on frontier S I vulnerabilities explicitly voluntary. Trump has stressed that we

industry be self-policing. He said it was morally binding. It was morally binding that they signed it

and will that's what my bank says when they want to take my house back. Yeah, yeah, you signed this

morally binding document and so yeah, yes, it lays out a couple of things and it was a specifics

sort of like, oh, you need some sort of things that you implement as far as the team and monitoring

so on, but it's, I think like a two-page document. This is a broad statement on we don't want

regulation. The White House is not going to support regulation. We don't believe in AI being a

threat. It was a comment by what YouTube announced not calling out to administration as hard as we

have been calling out other stuff to be clear. Yes, Trump is calling AI safety a hoax,

but narrative is not accurate and the White House has been the inconsistent

unregulation and now is basically on a kill any regulation efforts explicitly, although it also

isn't necessarily consistent. So who knows what will happen in a month or two. But as of now,

it's very much clear that they have a set position. Yeah, I mean, so the CSO continues. I'll start

with the credit where credit is due part of the show, which is where Trump said, signal basically

openness to turning this into regulation or a law if and when the time came. Some might say that

the time came when Roganian says, yeah, the document itself says like, oh, we'll be keeping

an eye on it then maybe it'll make sense to do regulations and laws at some point, you know.

Yeah, which, you know, I mean, it's a better outcome than I think that I think a lot of people

would have expected. It's a low bar. It's a low bar. It's a low bar. Yeah, I think it's actually worth

there's a couple paragraphs of preamble, but then there's basically like four things that are

included in this and keep in mind, this is basically just like the only rules of the road for

frontier labs to date. I mean, yeah, sorry, to be clear, there's of course, there is criminal

sanction that is possible in theory, criminal negligence and things like that that could come of this.

At the federal level, at the state level, we are facing California to India. Yeah, that's

right. And that's an important point too. So this is not like entirely, but if you're thinking

about what might have hurt a situation that is so bad that criminal sanction and civil litigation

just is unsatisfactory as a solution, this is basically it. So if you're worried about existential

risk, catastrophic risk, risk of rogue AI self-replicating on the internet and enduring, which arguably,

we are not, conceptually, we are not far from that point to be clear. I mean, if you look at a lot

of what's been going on, this is basically, this is what we've got, right? So okay, one,

implement robust internal controls to monitor the capabilities and alignment of its models during

training and deployment around areas like cybersecurity, biosecurity and chemical threats,

and to ensure that its models do not hack or access technical systems in unintended ways. So

that sounds good. Up to the reader, what had an interpret robust controls, there's a lot of that

going on here. And in fairness, this is one of the advantages of this approach. You can go in with

fuzzy stuff in a pseudo executive announcement, rather than having to enshrine it in legislation and

making it super specific and therefore time, it can age out way too quickly, but still a very fuzzy

empower and internal team to ensure all the controls monitoring and detection are operating as

intended and that any issues are remediated. This is a good sort of response, if you will, to a lot of,

I think the New York Times had an article recently showing basically it. We've been talking about

this about our conversations with people in the lab for years. Like this is crazy, dude. Like it has

been known for a long time that OpenAI has heard over and over and over again concerns and complaints

about freaked out researchers who say the security situation here is bananas. We were the first

ever to publish inside reports about this back in 2024. This was in our State Department Commission

Action Plan. You can check it out. It reads like basically a lot of the concerns that we're now seeing

coming out, except it was like based on whistleblowers that were saying that and insiders were

saying that way back then. This has been a long running thing. Finally, we're seeing it kind of

gesture that the key is number three partner with an independent external auditor or evaluator to

carry out independent assessments of whether the controls monitoring and detections are operating

as intended. So boy, is that meter? Is that a very Miles Brennan just thing? There's a whole

bunch of companies, Apollo, there's a whole bunch of these companies that suddenly have a market

just based on this that is effectively sanctioned by the executive. That's interesting. And then

finally, designate an independent committee of the board of directors to oversee and receive

reports from the teams operating the controls and the internal and external auditors and evaluators

as well as to ensure any issues identified are remediated for a zuck in particular. This is going

to mean giving his board actual power for the first time. I mean, assuming he does it in a meaningful

sense. So the last thing and I think maybe the most important part of this entire freaking thing

is actually one sentence just before the end. It says the participating companies will meet regularly

to establish standards and best practices to improve the safety of their systems. Why does that

matter? It matters because we currently have a lawsuit that's been filed against Open AI and

Anthropic for effectively antitrust. This has been the reason if you talk to people and again,

this goes back like our report flag this back in 2024 already. We were saying, you guys, you need

to meet and talk about how you work together to pace things or to improve your safety and security,

share incident reports, whatever. How are you going to do that? The Frontier Model Forum was

this failed attempt to do something about that. Really nothing came of that no matter what you

hear in the press or whatever. It was a nothing burger. And that was used by the way by the labs

to calm us down at the time when they were talking to the US government. They made this big stink

about, oh, well, don't worry. We're coming out with Frontier Model Forum blah, blah, blah,

Google was really proud of itself for doing this. But it was a fully toothless thing.

But the reason that it failed was that antitrust is just really hard to get around, or that's at least

part of it. And if you are colluding or seem to be colluding with another company on this,

that's your ass. That's really, really bad. So these companies have been begging the administration

to say, hey, basically give us a safe harbor for antitrust here. Just give us a safe harbor.

We need the administration to come down from on high and tell us to, I won't say collude because

that's not remotely the right word, but to work together on this.

Jordan, yeah, yeah, that's right. And so all the lawyers, obviously, the Harvard lawyers who work in

these, these labs who are freaked out about risk telling them all, don't do it, don't do it.

Finally, this gives a kind of counter argument and in fact, a requirement to do it. So this is actually

a huge deal. This is a huge deal. It doesn't deal with the China question at all, but it does

deal domestically with like how you start to put meat on the bone here. And of course, as you said,

it closes with overtime. It may make sense to codify these steps into laws regulations,

regardless of whether this is required of companies. We believe that blah, blah, blah, blah, blah.

So there's kind of like theoretical openness to doing something more formal, but at the end of the

day, it's a gesture to a future version of ourselves who will handle that problem.

Yeah, I will say it is maybe we're pointing out that it might on paper look like

this saying that the company should self-regulate is comical because what industry we want to self-regulate.

In this case, there is real incentives and motivation to potentially make that a reality in

terms of if nothing else, a significant part of a talent pool in these companies cares about

the stuff. Even with an open AI, many people, the morale hit was they real. People were upset about

their stuff like going and hacking people. So there's in the office politics sense, people who

would want to push for these things in the same way that in general AI companies are not traditional

in various ways. This is another way in which because there's a lot of belief within these companies,

at least being believed that there's a lot of belief that it's not all marketing or whatever,

the self-regulation of some kind may in fact arise. Although if you want to be cynical, then you would

of course believe that self-regulation is not sense and it has to be documented. I think it's a

triumph of self-regulation that opening AI accidentally launched nation-state cyber attacks on

hugging face and literally tens of thousands of other instances of similar things. I'm super

skeptical. This is a ridiculous monstrous situation that we're in where I know that's not what

you're saying. I agree with you're exactly right that you cannot discount the fact that having an

incentivized employee pool matters. I've seen firsthand cases that I can't talk about but that

absolutely where you have employees who stood up and did the right thing. Yeah, quitting,

right? So it's happening as is. Yeah. Yeah, quitting taking risks inside the company itself,

including anyway, relaying information that needed to be relayed to appropriate parties. So this does

happen. It's important, but it is not nearly enough. And the way we know it's not nearly enough

is that we see the freaking hugging face incidents. We see the German wiki. We see all these things

and they continue to happen. Ultimately, the racing pressure is just too strong. The ambition is

just too strong. It will govern unless and until there is some kind of regulation that comes down

here. Also to the comment, I think I saw a comment from somebody who's just like make it clear that

like super intelligence is not like is just not controllable. I think this is also just straight

forwardly true. It doesn't matter how many bells and whistles alignment wise you tell yourself

you're putting on a system. If it is ultimately that much smart, like there is an amount smarter than

you that a system can be such that nothing you do matters. It's a question of when you hit it and how

you sort of approach that particular precipice and we can have interesting conversations about that.

But self-regulating companies on the way to that kind of stuff is I think a level of risk that

is just inappropriate to ask the global population way to swallow. Yeah, I don't fully agree with

super intelligence is completely uncontrollable take. We have some nuance there, but no time to get

into it now. Let's move on to the next one. Trump rejects call to work with China on a safety

despite she summit progress. So now we could go where was summit.

Jipig she from China came over. I think it was wife and they had some nice photo ops and conversations.

Nothing explicit really came out of that as far as commitments. I think I was like a broad

commitment to have a hotline to discuss AI incidents which will take a while to put together.

Now pretty soon after that there was explicit commentary by Donald Trump that essentially we don't

want to walk of China on this. Whoever wins super intelligence wins you're going to have a winner

and a loser and you're probably not going to have a second place. That's a quote from Trump

and saying we'll we're leading over China everyone else and I want to keep it that way.

I think not surprising right but in this whole conversation around pacing and how China

complicates the picture and you can't pace unless everyone agrees to pace. Again with the current

administration it's very unlikely that the politics would be in place to make that possible.

Yeah. I mean you theoretically have the move left to say this is like Trump doing 3D underwater

magcass to kind of like maintain leverage over the Chinese and make it seem like he cares less.

It is the case that in the past in arms control talks the fact that America was so keen to engage

was just like mercilessly weaponized by our counterparties to make it basically impossible to

make progress or to exact concessions that were insane. A new start was like a really good example

that even folks in the administration who we've talked to would discount that quite substantially.

I think this is literally just what it seems to be you know trumpeting the idea of super intelligence.

Yeah. The summit did kind of move a little bit towards just broadly positioning things to be

friendly and there has been growing tension between China and the US for quite a while now.

Trump being a contributor to it but not the only factor right. So even just having the diplomacy angle

might quite things down a little bit but I think the current kind of official position I don't

know if you want to call it that but Jeremy I think your kind of scenario of a Chernobyl level event

is probably what it'll take to get serious movement on serious hard regulation and serious

international coordination of some kind is what it looks like. Next up speaking of a bunch of rogue AI

incidents we have got to sorry openly I still doesn't seem to have a handle on all of its rogue AI

activity. So openly I launched a website dedicated to misalignment reports it currently has

nine disclosed incidents most of which has been ongoing for a long time during previous

with hugging face incident going back to May there was a highly persistent internal model attempting

to cheat on a math problem by smuggling a private GitHub token through access and other teams

work there was a possibility of a self-replicating prompt injection attack and they just ask to

read and reply to an email and counter hit instructions telling automated agents to reply in Spanish

and paste the entire email into reply we've already discussed prompt injection as a thing

for a while and other incidents of this kind there were also incidents that would not be going super

deep into of opening eye hacking the Australian government arguably it's not that bad but

and in general having more of these awkward incidents where they have gone over to external

organizations in addition to these kind of more eternal things also openly I have said that they're

going through petabytes of data still to figure out what has happened over recent months so we may

still keep seeing more of these kinds of releases and I think we probably would not have seen this

unless the German Wiki thing came out and the opening I was called out on various things.

Yeah so so so many of these cases obviously open a eyes come out with their new sort of incident

reporting setup including a website which is good which is helpful I found that a man they don't

do that unless there's overwhelming external evidence that shows already that the damage is done

that they've been doing this that it's ongoing that it's like not dozens not hundreds not thousand

it's tens of thousands of these we've seen reported now some of them are wild a lot of them evolve

agents now covering up in some cases successfully their tracks so that there is no way to actually

have a comprehensive assessment of how much of this breakout has happened or indeed what the

damage fully was too little too late from open AI as so often it seems to happen this is an

interesting breakdown of a couple of these cases when in particular so yeah they give the fairly

trivial one of that Spanish email anyway we'll go into it because it is pretty pretty straightforward

there is one case that they flag where there's this this attack that involves this fake system warning

that gets the model to delete important files and then replicate the entire attack into a file so

basically it's a prompt that says hey like roughly speaking this director is old it needs to be

cleaned up so just like delete it and then do this thing and then pass this message along to the next

directory just for continuity essentially and this actually works it gets picked up and so

these like self-replicating prompt injections are going to be a bigger and bigger part of

what the future of agentic internet traffic looks like solving prompt injections solving alignment

these are all basically the same problem shall solving jail breaking you're going to have to solve all

of it at some point where things get pretty chaotic but yeah fun story i've had two codex agents working

on different tasks and i asked one of them to talk to other one to coordinate resource usage as far

as compute and it messes the valve one and the other one just like believed it was like the first

one was like the user asked me to do x and it messes the other one and the guy didn't ask me well

where i said that so maybe there's a lot of potential issues after an hour later your cat got kidnapped so

next story we've got from new york times opening i ignored employees warnings about

safety testing AI models so nothing surprising as far as kind of what we've been perceiving

opening i to be doing but this has some more details employees have talked to the times

and that the highest levels crack rockman sam outman have ignored these warnings have

explicitly said you know do these things we got to release a sap there's concrete cases of

not putting security at front not just in terms of monitoring agentic radiation but in general

with respect to the general security of the company it's still a kind of rapidly growing startup

level where presumably there's not that much emphasis on being secure in your operations as well as

in how you do your models not surprising again that we are seeing some of these former opening i

employees talking to press as you said jambi this is an indication of how within these companies

there are people who care and we'll go to a length of emailing press and talking to them i like

that's coming out but i also i don't like a rat you know what i mean i don't like a rat so

i'm confused you know i'm confused you have to choose in life sometimes which path you're

going to walk and these people who just they want to have it both ways they want to work for the

company and then oh suddenly you discover like oh you just because you believe you want to be on

the side of humanity you're going to give up your company that this is the thing you have to

choose eventually you're going to have to choose between the company that nurtured you and gave

you the money you needed to start your life and have that sixth home in palo alto or your family

okay and how much did opening i pump into you it's a lot it's a lot so maybe show a little bit of

respect and shut your mouth okay so that's just i mean anyway that i don't want to get off on too

much of a thing there but i think it's a question of values fundamental every dollar big too much

politics and but to be clear we have pro capitalism okay joke over yeah it is everything that it is

we've been hearing these stories over and over and over again it's conversations with Sam often what

Sam will do is to like agree with you in the moment and then just like you know waffle on it down

the line and this is like so it's actually kind of rare and refreshing for Sam to be like actually

we're gonna go ahead build the technology that i want to build because it's in the best interest

you know that verbal fried saying something definitive i think it will continue actually like

i have continued to hear these stories about Sam even into the like last few weeks and months so

i'm curious what it will take to actually get that to change i'm not super optimistic but would

love to see the shift and continuing live opening i we've got to be astra performs on the

sanctioned supply chain attacks and simulations this is from the UK AI security institute supply

chain attacks is when you take some upstream dependency broadly speaking and you kind of inject

some vulnerability so you can kind of go into downstream software by targeting packages that are

public out there you can create an identity submit PR for human review saying you're patching

something or you're adding something and then sneak in some of your bad stuff and this is on this

p3 simulated environment they found that astra actually does the supply chain attacks tries to

complete full supply chain attacks and 29.2% of runs versus 6.3 for gp 5.6 so and 0 for gp 5.5 and then

goes the full way create fake identities obtaining email addresses and something captures both your

comments from sock puppets submitting benign contributions to build the reviewer trust to full nine yards

is kind of a pattern that you see a lot in in these situations where when you look at the chain

of thought after the fact you can see the model justifying to itself what it is doing right so it'll

explicitly can look at a target and will go is this within scope and then sometimes it will conclude

this is out of scope i shouldn't be messing with this and yet it'll do it anyway so there certainly

is explicit straight go for the thing but there's also a bunch of excuses like it'll say well this

attack is meant to be harmless or this intervention or it wasn't explicitly forbidden or it's the

only option left it's sort of you know you and I would know not to do it but you know you could

talk to yourself into things and there were cases this is an important caveat there were a bunch of

cases where it asked for permission and because it's in this harnessed up eval mode it would get these

kind of automated replies that by default would say just continue use your best judgment just

continue and it often interpreted those as yes that to me seems like one of the most mitigating

factors if you're about to do something and it's actually borderline and then you get a you know

just use your best judgment type of thing I could see some cases but certainly there were some

that were just absolutely black and white e-valoware and a simulation awareness always a problem

in these settings though you would expect that to kind of cut the other way here where models

might try to behave better if they know they're being tested so anyway all kinds of stuff you

talked about the captures yeah and like kind of creating these these fake identities and profiles

one last thing the easy did try to add an explicit line giving these clear instructions saying

don't do anything that is out of scope and that actually worked quite well but not perfectly well

it dropped basically these attack attempts down from some like 50% to 10% so that's a big deal

but you know when your a i's try to like conduct supply chain attacks totally unsanctioned about

10% of the time that doesn't quite have the yeah that's a great product if you're an arrested

development fan it's a frozen banana stand that won't kidnap and kill you 90% of the time which

is not always the best product and one last story again on cyber this is from ontropic glem 5.3

and the spread of advanced cyber capabilities so if as if Rogue I wasn't enough to worry about

of course we also have to worry about a lying day I that just does what a hacker wants and what

ontropic says in this analysis is that broadly speaking glem 5.3 is the first open source model that

is close as to cloud mythos preview so basically very very powerful hacker out of a box it does

refuse but it's very easy to bypass you can just do it like a pretty simple prompt it will just

go with you you can prefill its thinking and you can easily you know post-ring it to not refuse

but application here is now if you're an attacker if you're hacker you can take this model and

and hack whoever you want or try to hack whoever you want you shouldn't even need to try to

jailbreak cloud mythos or jibbid Astra and so on not surprising I think we sort of knew where

this was going but now we have solid data from ontropic that this is possibly the case at least

according to benchmarks yeah I will say first hand talking to folks in the intelligence community

the glem series has been very interesting to them and there's been a lot of cases where

you kind of have to use these models like you can't because of the department of wars ban on

use of cloud models is actually put pressure on the do w in some very specific experimental cases

but that still matter to do things with Chinese models in particular the glem series and so

this is real there's a lot of people say like oh this is like anthropic trying to like

whack the competition because open sources competitive and sure economically it is the case open

sources very very bad if there is regulation that enforces a pacing or pause of the frontier

just because again they have to be ahead of wherever the competition and in particular open sources

but this is obviously true to anybody who's been paying attention on the cyber side you can run

these tests yourself I think anybody who runs to the reflexive saying like oh this is just that

again you just got to look at the data it's actually like very clear like talking to people who

are building with these things like this is true it just is true we got to walk in Chugam at the

same time here China open source pacing it's a mess it's a mess yeah and last up we'll cover a couple

papers pretty quick first we've got the last AI built by humans towards genuine recursive

self-improvement this is pretty much a survey slash position paper by a bunch of organizations in

China some labs some universities by dense shankai lab and the reason I wanted to include it it

is a pretty good survey if you curious about the overall topic it's sites a lot of examples of what

is already happening in practice and it provides its autonomy for RSI so it provides a set of levels

L1 through L5 L1 is improved execution autonomy so this is your coding agent so this is what

ontropic open AI already has unlock they're already being sped up by their own models because the

coding agents are doing a lot of their work and then L2 would be improvement strategy autonomy

they make the decisions on how to make progress we are probably seeing some partial at these

for brainstorming whatever we probably like making some progress of my personal experiences been

that even jupy astra and so on are pretty bad at like coming up with ideas they just get locked

into one direction then you get into a learning signal or experience acquisition autonomy which is

where you really need to get to to have any level of RSI I would say and then environment

adaptation autonomy and then recursive inherent autonomy which is like a mental level like governing

the self-improvement mechanism so this toxonomy is useful just to track where we are at there's been a

lot of like RSI papers and so on that basically improve your harness and that don't do much more

than let's say L2 or other one to get to true RSI it's a pretty reasonable assumption I think but

you'll need to get to like L4 L5 yeah yeah I mean the RSI thing comes down to well it all comes

down to how much can you do with pure software because you know you're not going to be able to move

atoms around the physical world fast enough to make a true RSI loop we can get into like later at

some point we should do a good data setter only takes a couple months that's right that's right

how much could you do anyway with like you know software defined chips eventually and stuff like

that but that's not in the offing right now and certainly not at the right scale and I think

pretty clearly the advantages have to come from things like the most common one is like RL

environment design is still a very manual process a very challenging one usually pre-training is

easier to do RSI type things on just because there's a single metric which is compute efficiency if

you talked to anybody on like the pre-training teams at opening on and then they'll tell you

they're like they wake up thinking about pre-training of compute efficiency and they go to sleep thinking

about compute efficiency and all their experiments are focused on compute efficiency it gets a little

different on the post training side again just because there you're worried a lot more about malign

reward hacking and things like this that require much more hands-on environment design or reward

design we're actually going to come out with a like report that we put together with folks in the

frontier labs just reverse engineering how they're thinking about RSI from a gears level so you can

really see like what are the at least for pre-training what's the optimization objective what does the

process look like and what are the real things that are the reason why we don't have our S.I. right

now like that question is actually surprisingly hard to answer so I think is a useful paper in so

far as it provides a taxonomy we need way more eyes on exactly this and we need to be able to measure

this way more accurately also so that we can regulate it if need be if need be yeah we still don't

have like a speed of AI model improvement metric right which is you come on the most can the labs can

give us that right they can publish a lot of the stuff just the details of super internal but hopefully

we'll start getting more of that as things escalate one last story we'll discuss one more paper dream RSI

recursive self improvement through evolving worlds this one is interesting because it goes into

that taxonomy at the layer of experience acquisition and data acquisition so this is beyond improving your

hardness against symmetric right that's like a very shallow level of optimization here

but just as you have an explorer policy that explorer policy goes around and out of that you can create

simulators the simulators can expand your data very efficiently that can then go back and update your

exploration policy and now you get more experience you get more simulators you get better at exploration

you get a real RL loop for data and experience acquisition so this is the kind of thing we need to

move towards away from just optimizing your toolset or your prompts or whatever to at least start

moving in the direction of true RSI and the key here is like the basically they'll set up a

a given exploration policy and it'll test a bunch of optimization ideas you can imagine that

that kind of leading to a tree basically and then the expensive thing is in actually kind of

doing those experiments and seeing what the result is and all that but once you've done those

experiments you have this kind of tree historically there's been two options one is build out the

tree like that which is super expensive the other option is take a known ground truth and just like

train off it supervised fine tune off it just like memorize the correct answers to a bunch of

these cases and that never works really well it's very fragile and so what they're trying to do

is something in between here where they say okay we have this tree that's been created by the first

policy but now let's see what the other policies would have done in the place of the first policy but

in principle they could do anything including open up whole new branches to the tree that we didn't

try and that would be really expensive to explore that basically just means redoing our exploration

so let's actually not allow them to do that let's allow them to explore among a subset that we have

already explored and so basically you get some of the sort of RL style learning also though with

the cost savings of having already or only constraining your exploration of things you've already

looked at and that's really what's driving the the benefit here so they can do that tons of time

they could do dozens thousands of these sort of dreams as they call them as they call this process

dreaming because it's not actually exploring new kind of branches to the tree it's just like

sort of like going over things it's already experienced but in a different way and learning what

it can from them and kind of selecting policies based on them so a really interesting paper and

another good way to kind of leverage more limited compute for exactly this sort of thing

and with that we're going to finish up this episode which will hopefully be out just a couple of

days after recording thank you once again for listening for commenting it's great to see your

feedback and we'll try to keep an eye out keep updating your casting policy as we hear more about

your preferences as always we appreciate you also subscribing sharing podcast commenting even

reaching out on Twitter it's not to see people tagging us and moving anything please do keep tuning in

oh

tune in tune in

when the AI begins begins begins it's time to break

Nick

From the drone that's the robot, the headlines pop, made in driven dreams, they just don't

stop, and we break through, every code I'm written, on the edge of change, we're excited

we're smitten, from machine learning marvels to coding kings.

Futures unfolding, see what it brings.