#252 - GPT 5.6, Grok 4.5, Nemotron-Labs-Diffusion, AI 2040

Hello, welcome to the last week an AI podcast we can hear chat about what's going on with AI as usual in the

episode we will summarize and discuss last week's most interesting AI news. I'm one of your regular

co-hosts, Sandra Kerenkov. I studied AI in grad school and now work the startup

aspect. I'm your other co-host, Jeremy Harris. I'm wearing a hat because I got back from taking my

daughter to the park. And yeah, actually, I'll be away by the way for the next episode because I'm on a

trip to the States doing a bunch of AI security stuff. But excited to do this one, we punted the

recording from Wednesday this week. We're now recording on Saturday and we were just talking about

how a bunch of stuff came out in the interim, which is really great. Challenging a little bit because

the way my schedule works out, I'm not able to like spend the same amount of time between Wednesdays

and Saturdays. So I'll be a little lighter on the details for some of these things. But I've still

tried to prepare a bunch of material for it. So be warned, Andre is going to be carrying some of the

weight for some of the more recent releases. But yeah, hello, a lot happened. Yeah, it's one of these

things where it's like a thing in the industry where we get waves of releases because I guess everyone

wants to be one-appling each other. And everyone kind of knows when each other's stuff is about ready

to go. So everyone just sort of like preps with hair releases and announcements and then they just

do it around the same time. And that's exactly what happened in this episode. We'll be talking about

GP 5.6. We'll be talking about GROC 4.5, Muse Spark 1.1, a few other models on top of that. And

beyond that, the big story I think will be in policy and safety with some new vehicle research

from on the topic and a new kind of major thing piece from AI 2040. So it should be a good mix of

content in this episode. We'd like to thank Box for sponsoring us. Box is building the intelligent

content management platform for the AI era. So we're going to secure essential context layer for

Box's AI agents to access the unique institutional knowledge that makes a company run. And that's

the key idea. The power of AI doesn't come from the model alone. It comes from giving AI access

to a right enterprise content. Box's recent state of AI in the enterprise report found that 96%

of organizations say agents need access to company specific content, but only 36% have connected

the agents to trusted content across many use cases. And a trust bit is very important. Box is

built of security, compliance, governance, and threat protection in mind. So employees and agents

only access information they're authorized to use. If you're thinking seriously about adopting

AI in your company, think beyond the model. Your business lives in your content and Box can help

you bring that content securely into AI era. Learn more at box.com slash LWAI. This episode is

brought to you by Outshift Cisco's incubation engine. Today's AI agents operate in silos,

limiting their true potential. We've been focused on building bigger, smarter models,

but scaling up is just one approach. To reach super intelligence together, we need to do more. We

need to scale out. And we actually have a blueprint from 70,000 years ago. Humans didn't just get

smarter individually. The cognitive revolution transformed society because we began sharing knowledge,

goals, and innovation. Agents are now at the same inflection point they can connect, but they can't

think together. That's why Outshift by Cisco is building the Internet of Cognition,

transforming AI from isolated systems into orchestrated super intelligence. We're creating an open,

interoperable infrastructure. Outshift is enabling agents and humans to share intent, context,

and reasoning. The cognitive evolution for agents is here. Explore the Internet of Cognition at

outshift.com. That's outshift.com. Let's kick it off in tools and apps. First up, we've got OpenAI

rolling out GPT 5.6 after government greenlight and announcing chat GPT work. We've now

publicly rolled out GPT 5.6, including Soul, Luna, all the variants of the model. I think we

already discussed previously how the model was. Soul was a little bit confusing in terms of

its benchmarks, but generally seen as like fable-ish, frontier, competitive, well-optropic,

and I suppose the bigger news aside from these models being public is this chat GPT work thing,

where they are updating with chat GPT desktop app, which used to be sort of just mirroring

website where you have a chatbot interface into just codecs basically, into an agentic code,

and then they kind of hid the actual chat functionality very kind of subtly in the UI. So they're

trying to onboard whoever is using this desktop app into using agents like hardcore, right? Which

is an interesting strategy by them and also rebranding to chat GPT work from codecs, which

seems pretty smart given like everyone knows chat GPT. I think not many know about codecs.

Yeah, and I think in and amongst all this to the whiplash on the model release permissions,

I will call them permissions from the US government because that is what they amount to is pretty

wild. So the US government just greenlit the release of GPT 5.6 of course. The request that they

had made was for a review period for the US government to be able to look at GPT 5.6. It sets

its capabilities and clear it for release. The strange thing that we're getting is there's a

report out in Axios that said that the government's position on the GPT 5.6 sole release was that

no permission for model releases is required or granted with respect to 5.6. And the decisions

about the scope of these releases quotes rest entirely with the companies. Now in theory, of course,

this is maybe the case. There is no law on the books. There is no regulation on the books.

Officially they could have just done it and then who knows what would have happened, you know.

Here's an excerpt and I'm pulling this off from I can't remember who's who's tweet I'm getting

this particular excerpt from but it was really I thought good sort of context for this whole

discussion. So if you look at the letter that let Nick sent to Anthropic when he was approving

fable going back on the market. Oh, kind of weird thing to even say in this context, isn't it? How

is let Nick approving fable if there is no permission required or granted like what anyway. Here is the

wording that let Nick used at the time. He said Anthropic has committed to work with the US

government on protocols and standards and releases for the quotes covered models. He says I

reserve the right to reevaluate and adjust the scope of license requirements on the covered

models should circumstances. What and this is the secretary of commerce by the way, not like some

sort of oversight thing for AI at all. It's it's very out of hawk essentially all of this stuff.

Sam himself said of the 5.6 sole release says bad news. This is on Twitter at the request of the US

government. It's it's launching today in limited preview instead of the open access launch we're

planning on we're working with the government to get to general availability as fast as we get.

This is not sound like a company CEO who has a free hand to do what he wants. It sounds exactly

like someone who understands that in reality what's happening is the government yes will allow

you to do it, but then they'll smack you back by having the BIS export control to crap out of

your company and effectively grind your operations to a halt with some of your most important customers

on a win. That's the reality here. So there's sort of this funny thing. I have no objection to licensing

regimes. But like we again we put one together like three years ago when we predicted this moment

would come. It's been obvious we were get to a moment like this and frankly a lot of people in the

Trump administration in particular have spent the last many years saying that's ridiculous. It

will be irresponsible. This is craziness. This is doom orism all this stuff and yet here they are

doing exactly that ostensibly but just not calling it that which I think is just an intellectually

dishonest play. I'm being quite forceful on this because I think we need to force a conversation

at the legislative level to have actual regulation do this with principle. I'm not saying any answer

here is right or wrong but it's like it is just obvious to all concerned that the current

approach is to wildly untenable and just false. We're getting like lies that are lies in practice

if not in theory. They're lies in practice when you look at like how is this stuff actually being

regulated. There is a licensing regime today in the US for Frontier AI full stop. You can dress it up.

You can say that it's just like oh no it's like this but like ultimately it's also inconsistent.

Opening I was just asked to I guess just to show you deference to the feds whereas anthropic

was export controls different mechanisms are being used here in what seems to be a fairly ad hoc way

and they're being very weekly and poorly justified. So anyway there is also all kinds of

undisclosed model loopholes by the way. So if you if you have a Frontier model that's built by one

of these labs and it's just not announced some of those models the internal deployments if you're

concerned about loss of control at least or weaponization the internal deployments are as dangerous

as the external deployments for the like a huge fraction of the risk surface China can steal the

model you can lose control of the model or rogue insider threat in your company can weaponize

the model like none of this changes and yet there is literally no effort being expanded seemingly

no requirement to disclose internal deployments. So not only are you seeing a totally unprincipled

uncoordinated seemingly semi thoughtless approach being applied here that's rattling markets

causing all this confusion but it's also failing to grasp that there are that the exact risk they

seem to be concerned about which is AI super weapons still exists like fully for a wide range of

the kind of threat vectors that you'd be concerned about. So anyway I just think this is like a

really like class example by the way is say super intelligence right a lot of people think that they

have something close to Frontier potentially if that's the case I mean totally undisclosed they

don't even plan on launching a product they may have these things running around on their servers

China all in there rush all in there whatever yeah who the hell knows right so this is just I mean

it's going to get fixed over time but there's a pretense here that there is in any way a kind of like

thoughtful approach to the line that I think is just laughable at this point this is a a chaotic mess

and it needs to be fixed we need to be able to look back like literally just I'm just asking for

two weeks of look back where the government looks half coherent in a two-week window please just

like give me two weeks where I can go back to look what like what did the Commerce Secretary say

and okay yeah that that's a line with what we heard the White House you know what I mean like we're

just not getting that this is this is just crazy talk at this point wow Jeremy it sounds like

you're getting to understand US politics I mean what I what I will say is it used to be more

coherent like when the White House put up the action plan you know just roadmap they more or less

stuck to it for like about six months or whatever and then the anthropic thing happened and I feel

like everybody just like got brain cancer or something and now like why I have no idea what the hell

anyone's position is on anything and they keep saying stuff that's like dude like do you actually

have a preferred lab because it really sounds like you have a preferred lab and that's not good it's

bad for your standing in the courts when you inevitably have to litigate this stuff it's bad for

the markets it's bad for the coherence of your security and security argument this is really that

so as usual we had to touch on the politics of this and one more thing to say on that front is

an interesting like as a comparison to what happened to an anthropic I saw after this release

that UK AZS which we've talked about I think quite a few times there was a post saying that

when they got access to evaluate GP5 Seoul they were able to get universal jail breaks within hours

testing which is what was not shown to be the case with an anthropic supposedly the problem was

like this very directed single jailbreak that kind of wouldn't give you much versus universal

jail breaks is basically we can make this model do whatever you want either way it's very clear

there's fewer safeguards in place than what an anthropic had anthropic had a very restrictive

set of things even with the initial launch that would like roll you back if they detected that you

were doing cyber stuff or whatever to my knowledge there's no or they are still cyber things in place

by opening eye but I think if you compare them it's it's very different so good for us to get GP5.6

so we got to vibe check I guess of a community and it's a good model people like it it might be

fable level it might be better than what an anthropic has so it's very nice to see the outcomes

of competition and opening eye sort of beginning its focus and getting back to providing a challenge

to an anthropic which we have been sort of focusing on other stuff for a while on that note right if

it is fable class one of the things here that is also pretty bullshit is that anthropic was held

back on their release and therefore their monetization of fable for weeks meanwhile like I want to see

the side by side of how long open a eye was forced to delay 5.6 if the gap is less that basically

the US government just handed them weeks and therefore hundreds of millions of dollars that's what

those weeks bring in in profit and so you know this is just another way in which this unprincipled

approach of seemingly I think to many people picking favorites almost shamelessly when you look at

the language that's used by it just some of these like it's crazy anyway so yeah it's crazy if you

don't live in the United States that this could happen but why we keep talking about it is that it

appears likely that this is just gonna be the situation going forward so this is not a one-off case

of it happened with these two models it looks to be the case of the administration is at the

point of saying okay there's gonna be these powerful new AI models and we have a say in what

these companies do and we can do whatever we want because there's no rules in place or no policies

and so on and so on so we can expect to keep talking about this unfortunately as new models from

on topic and from OpenAI get announced yeah just to also like you know people listening you know I

have been a defender of this administration's approach to AI for quite like basically for the

entirety of their first year at this you can go back I mean I think to a fault in some cases but

you know it's trying to find the logic where it was you know in Silicon Valley there's an

unfortunate bias in the other direction and I think that needs to be corrected for so you can

sort of see the signal and noise I think the challenge is this was all kind of cozy as long as

an ideologically aligned entity was in the lead I think that mythos just like broke everyone's

brains I think there was also you know David Sacks has no business having an opinion when it comes

to AI like Mark Andreessen these are people who fundamentally like do not actually understand the

technical side of the safety argument and the security pieces and I'm like you can go back and see

like individual things that show that but you know you can hope that there is enough and I think

overall the same reminds will prevail to be clear like eventually it becomes just like these are

bio weapons these are cyber weapons like no one can really deny that this is a thing and I think at

that point you're going to see Congress step in you're going to like AI 2040 and all that stuff is

making is calling the very easy prediction in that respect that we've been talking about that for

years on the podcast alone and so sorry I mean on the podcast alone as opposed to other venues

so anyhow I just think we're going to get to sanity it's just like

people have to be moved different distances to get there let's say it's a bit of a mess with this

whole thing but if you don't want to hear what the politics I guess this is good news because

GPD 5.6 is out they have sole Luna and their small model and it is cheaper alternative to fable

while you know it's some first person accounts being potentially better and one of the weird parts

with these frontier level models is they do seem to be diverging a little bit in their flavors

of intelligence so these models always had a sort of character to them and slight quirks and

differences but in terms of their capabilities they felt similar to me and it now feels like

there's this high level differentiation between GPD 5.6 and fable where when doing very advanced

problems of like math and so on they have different sort of high level pitfalls and ways of reasoning

and looking into problem it's quite an interesting phenomena next up another new model this is

GROC 4.5 from space x ai so they are calling this an opus class model and if you look at the benchmarks

it's you know generally competitive opus 48 terminal bench deep SWE versus their coding model

alternative for both GPD 5.6 and what on froth I guess providing and the big news I guess

besides that they have a new model people saying this is a pre-trained model so this is like

cursor jumping in and training a brand new GROC that is much better at coding it's very cheap

relative to what the alternatives are there is the pricing is $2 per million in potocans and $6 per

million out potocans that's I would have to check but maybe two and a half times cheaper than opus

if I remember correctly and I don't know how much cheaper it is than fable so it's a relatively

very cheap model that appears to be pretty capable and from what I've seen of people's conversations

people seem to be saying that it is pretty capable in practice not just on the benchmarks so I think

we'll talk about some other model releases too if the trend is definitely moving toward a price war

where token prices are going to be pressured down especially now where fable you know you can use

an opus class model for your everyday needs like the frontier frontier models you probably won't

be using as your default and that means that this price war might get nasty you know yeah it's

actually quite interesting to think about where exactly you're going to see the profits accumulate

for that reason in the space like there's this argument that I think is implied maybe as mid-made

explicit I just haven't heard it quite that way people say there's infinite demand for intelligence

yes is there going to be infinite demand for the frontier of intelligence or does the frontier of

intelligence end up getting consumed by the labs themselves in pushing recursive self-improvement

like what's the shape of that pyramid basically I think that's a question we we don't yet know the

answer to and it's I think more non-trivial than I think a lot of people make it out to be myself

included myself from like three years ago certainly included so we'll see but the price wars are

are definitely going to be an an issue increasingly I do think the frontier will continue to command

a premium and I do think that it's going to be hard for players like rock to actually end up

leapfrogging anthropic we're seeing a lot of stuff like that some of the benchmarks that were

shared originally showing things that you know may have worked their way into the training

corpus that were like you know these cursor oriented benchmarks and things like that but what

interesting note about this too so we don't have unlike previous rock announcements well we don't

have as a model card that looks at all the all the usual risks and so 4.5 references a bunch of

benchmark comparisons but we don't have this like structured principle assessment of how you know

how things are going to work on the on the safety side we know there are some cyber security

safeguards that were added they're kind of hand-waved and I mean ultimately this kind of

hand-waving happens when you're concerned about US government stepping in and saying hey like you

can't actually release this man of which there is plenty of precedent now so anyway it's it's also

meant for like these long horizon tasks right with nominally full autonomy so the stakes do go up

whether it can actually execute you know on the kind of meter evals tight plots that that everybody

cares about on cyber and on AI and autonomy remains to be seen but kind of interesting it does seem

it doesn't have at least some pretty good roughly perito capability so it may not be the most

capable but when you look at capability and cost there's a point on the curve where you could

quite plausibly put rocks for 0.5 to be fair in the announcement post we do have a single

sentence saying we've also added new safeguards reflecting the models cybersecurity capabilities so

I think they've addressed it pretty well with that one sentence that's a fair point I would track

everything and to correct my previous statement it's one for for price of opus and one a for price

of fable so much much cheaper than the models from on fronpic also cheaper I think than GPD 5.6 so

very interesting point in time where we haven't seen getting it to his price for territory I

think until now and in fact we'll with the next model we'll have you in more to say so moving

right along we've got meta and they have released me use spark 1.1 which is essentially just their

update of the model that is good at coding or at least like somewhat good at coding enough good

at coding to make a claim but you can use this as an agent and potentially part of your workflow and

they are also jumping into a price wars I have to double check with prices but essentially it's

very very aggressive pricing relative to on fronpic and open the eye it makes sense because nobody

would even try this as an option unless there's a strong incentives in place and I've not seen

many vibe checks of people trying it but the benchmark numbers are looking fairly good and I think it

would not be surprising if in fact it is sort of starting to compete at least with open source

models like GLM and deep sea just given you know big collect data they have in for a big

should have gotten to this point given them out of resources they just shoveled it to his

yeah it's it's pretty interesting when you look at like the broad finger print of capabilities

is advertised here so the first thing by the way from a you can tell what part of the model

announcements I rush to immediately when I see these cyber capabilities because that's actually

going to determine how much money you can make it turns out in this current domain and by the way

don't think the bio isn't coming soon for in exactly the same way for exactly the same for

even actually worst reasons because you can't patch a human anatomy anyway so they do these

dangerous capability evals they say they can't rule out the high risk threshold for both chemical

biological and cyber security and they say that when they bring in their mitigations they can

get the risk down to moderate or lower which is you know the kind of acceptable to release range by

their assessment when you look at cyber though the capability jump is huge so side bench which is

the years one of the classic cyber benchmarks the score goes from 65 to 93 percent that's insane

like and remember right going once you once you're pushing 85 percent or even 80 percent on a

benchmark depending on what it is it gets a lot harder to score that incremental point so getting

from 65 all the way to 93 this is a really really big jump and this is relative to the previous

muous spark one so this is true basically for all benchmarks this with respect to coding all of them

have very very significant jumps I guess cyber being a part of that so it kind of tracks and to your

point they did release a hundred page safety evaluation reports so they do seem to have they are

safety framework actually being put into practice it's weird to see meta doing more safety work now

than grok the then then then space x but that's where we are yeah I mean Alex Wang I guess is

scale pill then and that's why he's putting it out but yeah and then so there's also cyber gym

which is more of a real world vulnerability discovery benchmark and that's interesting because

so 59 percent this is a score that mues 1.1 gets on that benchmark that is well below opus 4.8

opus 4.8 is like 79 right so like huge huge difference like 20 points G5.5 is like 82 so it'll

warn these cases you see kind of very uneven footprint here certainly relative to the other

frontier models it's impressive like you said and I think it is you know you can think of it as in

in some domains probably GLN 5.2 ish kind of adjacent which is impressive crappy today is

impressive two months ago like so you know it's worth it's worth padding them on the on the back

for this but yeah roughly GPT 5.5 comparable on bio so anyway all which is to say I think meta is

going to enter the chat on the regulated AI model stuff if this trend continues probably some time

in the next like three months I would guess so you know this is going to be an industry wide thing

it's no longer going to be just anthropic and open AI the president of the United States is going

to have to stare at the markets and say not today and and at some point that's going to have to

happen and there's going to be a big market correction I say that as somebody who is in various

ways invested as an angel investor as an investor in hedge funds as an investor in various things

I have stakes on this but I have obviously more on you know our survivable species so I hope

president is able to say no to the markets when the time comes and it's going to be a really tough

I mean that you know then there's the national security picture and that is a huge challenge like

it's it's not you can't say no to the national security picture you need to say how and that's

a conversation for another day yeah and we're noting too that I think it's fair to assume that

there are relationships in place and some people have more leeway and I would imagine it's true

that meta as one of the big tech companies with tech in general having made a big turn in Silicon

Valley it used to be the case that tech was progressive tech was sort of aligned with the left a

little bit that has changed tech is aggressively aligned with whoever is in power and I'm very happy

to work with Trump and basically cozy up to him and gain favor so I would expect meta in a way

to have a bit of an advantage with respect to getting permission to not have oversight and so on

and so on going back to a pricing so they are pricing 1.25 per million input 4.25 per million

output so cheaper than rock 4.5 even what is this one six for price of opus 48 getting towards

a territory of open source models I think potentially even at the same level which is used to be

like super cheap so again part of this price for trend yeah and one last thought too is there's

always this question of what the personality going to be like once you have a new entrant in the

frontier kind of space for potential one question obviously comes up like okay Groc is going to

you know do anything for you Claude is well Claude and then chat gbt is chat gbt it's going to

write you some lists the tradeoff here seems to be or the personality tradeoff seems to be towards

over caution at least that's one part of the personality here so the false refusal rates on

benign requests are higher than peers considerably in some cases especially on things like cyber

higher than gbt 5.5 and opus so it definitely it's tuned conservative not politically but like it's

tuned to refuse to answer questions when there's any doubt so I find that kind of interesting this

is like a complete pivot from like the Yanlecun era of Facebook meta to now Alex Wang the

starts to look a lot more like a I don't know like a cousin of anthropic that's a real stretch

but there's a flavor of it here and you let's speak yeah meta we had some other news as well they

introduced muse image and use video their new synthetic media generation models a bunch of

screenshots and videos that showcase that these models are quite good and potentially competitive

with a frontier also benchmarks showing with their quite strong and the muse video was just in

preview muse image was actually rolled out and made usable across their platforms resulting in

backlash because one of the things you could do with muse video muse image was use it directly from

I believe it Instagram and if you simply tagged someone who has a public account you could make

AI generate images of them like it was made incredibly easy to make edits of any account on Instagram

and the backlash was strong enough that they backtracked through release two days or three days

after it came out so yeah kind of a fumble on meta as part here where they appear to be making good

progress muse image and muse video as far as I can tell are quite good synthetic generation models

and this is a place where you know there's Google and Open AI competing but anthropic doesn't focus

on this stuff and not a banana is a little bit old chat to be T is not updated so they could get a

real advantage here given the amount of data they have of Instagram I would imagine that just the

training volume is ridiculous but the product side release here was a bit of a mess and next going

back to models there's a story Chinese AI models gained ground with the US companies as cost

search so this is a bit of a trend check where it's showing that the share of tokens that are

coming from these open source Chinese models deep seek and see that AI has gone up significantly

it has now gone up above 30% weekly since February 2025 on open router open router being one service where

you can kind of query and and direct your queries to different LMS and and there's some data there

we don't know how many requests are hitting the official APIs so again another trend of

what we're seeing where GLM 5.2 is incredibly cheap it's still I don't know if it's cheaper than

a meta spark 1.1 but it's you know same territory have much much much cheaper than

anthropic and open AI while being very capable GLM 5.2 had a very positive vibe check from

a community that you can basically use this as your cloud code driver for the most part and it is

sort of like one 10 for ish one 10th ish for price so we have some data here from open router

showing that it is starting to get effective open router of course is a bit of a niche you don't

know how big the impact is but it's happening yeah keep in mind also that China is going to be

clamping down on open source at some point in the next like year or two so many of these trends are

I think properly viewed as transients where it just becomes untenable for you know like cyber weapon

models or models that can even be used just to amplify the freedom of the individual if you're

thinking about China in ways that the CCP is not going to like you are going to see clamp down

they'll be justified in various ways they're not always going to be justified it's like oh this is

you're clamping down to clamp down but the effect is going to be that but in the transient this

is actually really important like if you're trying to think about an offensive cyber campaign as

part of the Chinese state and you want to like get all up in people's systems in the west we have

agents like AI agents that you can make sleeper agents and you know our ability to control those

those agents is only an increase over time and so while you have Chinese agents from deep seek or

you know GLM 5.2 running on your infrastructure like that is an insider threat and that is a way to

people need to start thinking about it and I mean I know that they are in various quarters but

increasingly even at the corporate level I think people are going to start thinking about it that way

sounds like science fiction but like that's just where all the scaling lines are pointing so I think

it's an important flag that like just in the same way that people are suddenly get feeling a

shock from over alliance on the frontier labs and like oh man the US government could just like shut

down my access like this is no good yes they can they will continue to do this like until morale

improves the government will keep clapping down on models but the same is going to be true and

it perhaps ought to be true from the open source side increasingly over time you know if you if

you don't want your your like IP your commercial secrets to be like sent overseas increasingly you're

going to want to like look at a model that was made you know with the hardware that you trust or

separate trust yeah a couple more things I'll say on this front I think it's a nuanced picture and

I think it's interesting we use open source models because they are open source so you can

take the actual weights that are public of the model and just look at their performance and this

is why having safety institute and cyber kind of protection institutes is actually very beneficial

I would imagine is if there is a sleeper agent component well to be quite the big failure of

the safety institutes to fail to detect this potential in these models and secondly because they

are open source I mean you can align it all you want after with just some post-training right so

the picture is a bit modeled is all I'm saying is because they are open source the models don't

have to be used as is once they hit the west and they are providers on the western front of just

inference that you would probably go through rather than going straight to z.ai and so on or try

to host a model yourself you would go to fireworks or grok and they typically put these models on their

platform with some caution so it's a bit of a new aspect and we can expect for the near term

for these models to keep coming out and keep providing pricing pressure absolutely on the

pricing pressure and I very much agree with like your structural assessment right like having

the weights definitely reduces the risk considerably one of the challenges is like we start to

flirt with we basically need a solution to the alignment problem at a certain point because we still

don't know how to even detect sleeper agents let alone you know excise them from these models and

there's like maybe that'll change but there are a lot of people in the alignment community who

actually see the sleeper agent kind of intervention as being equivalent to a big part of of solving

the larger alignment problem which a lot of people don't think is theoretically possible but like

whatever so we fork out and you're right like there's a world where we certainly benefit either

way having access to the weights how much we benefit is a really big challenging thorny open question

and I think it ultimately it's a leading question the nation states will be asking themselves as

they inevitably start to weaponize these models like it's it yeah this is going to be part of the

future of sub threshold warfare yeah we have seen by the way I mean this is not just

speculation we do know with these models if you ask them about things that relate to state

sanctioned discourse if you ask about Chinese history the models are going to give you the Chinese

history from the government's perspective in every case so these models are very much already

being influenced by a policy and it's not yeah I totally agree that there is a real potential

for them to be you know the sleeper agent thinks sounds ridiculous but it's it's possible right

onto applications and business and first we got meta again when use is there planning a cloud

business to sell AI computing power this is according to Bloomberg they are doing this cloud

infrastructure business internally called meta compute that would sell outside customers access

to AI computing power and men as own AI models competing directly with AWS Microsoft Azure and Google

Cloud very interesting kind of mirrors what happened with XAI or space XAI now where they were a model

company and and more so beyond being a model company they spent absurd good gillians of dollars

building up data centers that were kind of leading cutting edge and they built as much computing

capacity as was possible basically as quickly as possible and were they able to use all their compute

who knows it's actually you know to ramp up your team and your training processes and the

infrastructure and so on and so on having four data centers suddenly doesn't mean that you can

utilize all four data centers effectively and I think where is the case to be made that meta

as an actual business that needs to make money and doesn't get cagilians of dollars from VCs

is starting to look at their balance sheet and saying we are spending cagilians of dollars

on these data centers and what are beginning and return nothing and that would mean that there's a

real you know case for them to seriously move in a direction I like the word cagilian has now become

a unit of measure on the last week in AI podcast we've we've given up now the economy is unmoored yeah

there's a lot going on here one piece is it is absolutely true that meta so every company has

this space X I'll call it problem or incentive where if you're trying to compete at the frontier

you're going to try to buy hundreds of billions of dollars worth of data centers a year and a half

from now you're going to build them and then you wait and if it turns out that you fucked up and you

don't have a genuine frontier model because your space X because you're meta because you're whatever

now you have a bunch of excess capacity and like you don't have an economically valuable use case for

it what is an AI model it's a funnel that turns compute into intelligence that's what it is and so

there's two kinds of value there's the funnel and then there's the compute and then you put those

together some companies have won some companies have the other so meta here does not have the

intelligence they don't have the frontier models but they have the infrastructure so yeah that pushes

them to towards becoming a new cloud but the challenge they have with space X is that space X is

differentiated by its institutional and cultural capacity to build infrastructure really really fast

meta has historically not been scale-pilled it is historically not been the company that believes in

super intelligence through scaling it is historically not been the company that like makes bets that

or of the same shape as the anthropic opening I bets and so I think like right now a lot of

the stories framed around excess capacity that meta happens to have right now it's unclear whether

they're actually going to like pivot in a long-term capacity and the same question by the way has

been raised about space X like you know colossus one they rent that out colossus two you know they give

it yes to cursor but then they acquire cursor and so like yeah it's kind of in house maybe they

are trying to get that centers of space right that's interesting yeah exactly like so and the question

is like whose model runs on an infrastructure and what's becoming pretty clear and that wasn't at

least certainly wasn't obvious to me like three years ago was that we may end up living in a world

where you have competition among the frontier labs any given time to figure out who's got the best

model and then like once it's clear that let's say cloud is the best model now everyone re-oriented

their inference compute to service cloud because that's just like the most financially like the

highest ROI thing for the from the market standpoint it's just rational now everyone who wants to be

a frontier lab continues to compete along the training compute axis because they still they want to

get that best frontier model for the next generation a two weeks later we want everybody to be

competing to run our model but you could imagine at least the incentives one part of them that is

legible to me at least right now points in this direction where you actually might have more

fluidity everybody's competing to build compute it's unclear who wins but whoever wins

they'll be happy enough to like have their model run on other people's infrastructure and rent that

and everybody else will be happy to to amortize the cost the ungodly i-watering cost of this

infrastructure like people aren't going to pay us to use our shitty model we might as well like

get money from the leading lab so that this whole thing wasn't a giant waste and so the question

really is where does that margin go from the actual model training part of the stack and that's where

you know for a while whatever company happens to be in the lead at that time gets to just earn a

lot more margin and so i i think that's potentially part of this you could ask yourself like what

does meta have over a typical neoclod in the space and like the answer isn't great they don't have

like jeep the tpu they don't have like these massive like internally developed really high performing

asix what they have is financial capital they're less risky than a core weave because they don't

have like crazy debt to equity levels they have an actual cash flow business in advertising

we can massively support this and so if they wanted to literally just because they're a floating

giant ball of cash with a decent technical bench to them they actually could move in this market

it's just a question of like how scale pill they are and where they see themselves stacking on the

kind of model part of the stack yeah i think i'll say meta to me looks like if i have a little bit more

of a advantage in the sense that their mature company that has data centers for their websites

right so they are are in the data center business broadly speaking they have have you know

however many they they've built many of them so they have the structures in place to do that

and the talent in place and they also have worked on custom hardware for inference which

we've discussed i think it's moving along to a decent place the real challenge is they are not in

the cloud business and have not been in the cloud business so they're competing with google cloud

a ws which have been clouds for a very long time and have served gpu capacity for a very long time

one other thing i'll say is to your point of we've seen this play out earlier this year on

frropic had explosive growth with cloud code explosive you know i forget it's like 10x 30x

ridiculous in the span of months and they ran out of compute it was like a ridiculous situation

where suddenly you could barely use cloud code if you were on the twenty dollar plan or whatever

because you constantly hit limits and the model's got worse it was very very obvious

they just did not have the compute to serve a demand and they had to scramble to make a deal with

xai and so on and if the projections about gdp and ai adoption and the acupabilities

yeah hold where you know we have another 30x jump in demand in like a year which is plausible

if you think that cloud code work and so on will continue to be as good for professionals as they

have been so far the picture of being a provider avidanta center starts looking very appealing

because then frropic will have to pay whatever you're asking to continue to serve their customers

yeah you're right and it's it's sort of this game of chicken right that they're that they're playing

it's like it's a horrifying like proposition for both because while in throttick is absolutely

desperate to open their mouth wider to be able to suck up all the sweet users that sounds horrible

anyway well you know you know what i mean i love yeah let's pretend i didn't say that but like

just the anthropic desperate for that on the back end you're you're the let's say your space x

like you're sitting on these these data centers that you built you're sitting on a massive capital

outlay that you just put out and it's not making you anymore and it's depreciating rapidly exactly

exactly it's so so you're actually like in a game of chicken's it's like nobody wants that to go

on for very long if you don't make money from it in a year like you're screwed you messed up but

gps are outdated like yeah you wasted your money right you know if you have a gigawalk

you know a hundred billion dollar like a gigawalk cluster roughly ballpark is a hundred billion

dollars or so if you have that just like hanging out like it's lifespan is like I don't know what

call it like like 10 years or something like order magnitude so like yeah yeah why don't you go

and like light ten billion dollars on fire you're obviously not going to do that and so you're

going to make whatever deal anthropic wants the benefit but you're dealing with the loss of

version at that point and so anyway it's a it's going to be an interesting economic time that's

why I think that the economic forces pushing for this kind of call it like compute infrastructure

fluidity where it chases like whoever happens to have the frontier model in that moment the argument

I think is pretty strong I'm I'm very curious what that adds up to for the neoclub and stuff but

seems like it's there and I think this also adds interestingly to a price for a picture where

to do a price for you need to be able to drive down your costs yeah ideally to be able to make

this profitable and one of the interesting things about the product is it appears that they

have become profitable they have managed to make this business model work but with rather expensive

models the most expensive models on the market when you have more supply of data centers and this

is a speculation on my part but you know you would imagine that if you have more supply of compute

in general hitting a market and I think all these companies openly I bet all of them have made

very very deep commitments in KPEX to continue building data centers so I would imagine the compute

is coming online and we'll continue to come online in the next year or so and then you know it's

a staggered process so as you get more of these data centers finished up and ready to serve

demand you have more supply of compute and that means that tokens can be produced probably

cheaper and that means you can charge less and great now it can compete in a price war and we

like consumers win basically but you see how it goes it also it depends on so many factors

like how does dollar denominated demand track with the actual intelligence per token and then

and then also like can you grow the data center base fast enough for that increased demand what

Claude suggests is like actually in some cases you suddenly hit a point where the answer is no

and so this is like really it's hard because it's a function of what new companies you can create

as a function of breaking through those new levels of intelligence per token and also a lot of this

is like just almost literally impossible to predict this is what markets are made to discover but yeah

it could work out in either direction and lots of room for speculation

BDM markets discovering what prices should be truly a miracle of I don't know modern

business you're listening to this podcast so I know you've got a curious mind here's a helpful

fact you might not know yet drivers who switch and save with progressive save over $900

on average pop over to progressive dot com answer some questions and you'll get a quick quote with

discounts that are easy to come by in fact 99% of their auto customers earn at least one discount

visit progressive dot com and see if you can enjoy a little cash back progressive casualty

insurance company and affiliates national average 12 month savings of $946 by new customer

surveyed who's saved with progressive between June 2024 and May 2025 potential savings will vary

why does progressive work hard for truckers because truckers unite the world they unite kids with

their first drum sets and parents with ear plugs but truckers can't do this if they're not on the road

that's why progressive has over 360 heavy truck employees to help truckers stay on time and on track

quote truck insurance today in as little as eight minutes at progressive commercial dot com progressive

casualty insurance company and affiliates next story related to data centers as well like US

energy regulator sets ultimatum for data centers services FERC the US energy regulator issued

tailored shock show cause orders under some section of law to each of the six regional grid operators

under its jurisdiction directing them to justify or reform the rules that govern how to data

manufacturing facilities and other large energy users connect to the electric grid and this

is a very notable point Jeremy I'm sure you know more about this of at some point the data centers

are exceeding what the energy grid can provide in the case of xai they have which guys turbines

that are spewing pollution and so on that are operating to produce electricity so yeah what is

the picture here Jeremy yeah I think I think you hit the nail in the head it's you know a lot of

these data centers increasingly are moving to behind the meter energy generation which means they're

not on the grid and this is happening really as much as anything not partly it's happening for

political reasons because it's untenable to have people's energy prices go through the roof and then

you get political pushback and ball of law like no data centers in my backyard Bernie type stuff but

then also there are other advantages to it and so now yeah regulator the energy regulators like

like p.j.m. is an energy is a grid operator that recently had to basically order generators

to run at maximum output and bring idle power plants online because there's a heat wave

and they couldn't support the heat wave because of all the new load on the grid so like this is

having an impact where you're flirting with brownouts right now and you know if you look if you

plot out the rate of of energy consumption from data center demand it's like right now it's in

the single digit percentages of total US energy production that is only going to increase and so

you're going to put stress on the grid and eventually you're going to find brownouts become a thing

and there are all kinds of ancillary issues where the regulators now getting their feathers all

ruffled you know one example and we're doing like a pretty deep dive on this right now that'll be

public in the next couple months probably but one thing to keep in mind is like so a lot of the

infrastructure that the energy infrastructure that these data centers use is like designed to trip

on the same signals and so if you have you can imagine like basically a correlated risk where

one kind of signal on the grid basically shuts down a whole bunch of data centers at the same time

so you've got this on the one hand like super high correlated risk because this has been a

totally unregulated market in a lot of ways or in this way and at the same time you've got the

risk of brownouts things like that the regulators like department of energy is going to have to step

in they're going to start to insist on big changes and so again you know this is one of those

things where the past trends are real but also like we're going to learn a lot about the impact of

government regulation in the next couple months but wait does anyone remember climate change

that climate share is a thing and pollution and carbon and now I guess it's just me yeah not

helping all of this stuff yeah I mean you know and I think it's that they look you could make the

argument that like for almost arbitrary climate related problems having better AI that can help

us solve these problems is a thing whether it's anthropogenic or not also because like you know either

way if they the heat you know it heats increasing in shit like the carbon is science you produce more

carbon it's gonna make things worse and you know but just like you could find ways like regardless

of sort like you could find ways around it that are more creative with it like well well like it

like yes yes yes but it is funny how the the discourse is like yeah the point also is I guess if

you may have a route rush to add more capacity and both out of centers and so on you're not going to

get clean energy renewable energy to drive these out of centers and and that's already been shown

to be of a case Google meta etc have made deep investments in powering their data centers with

renewable energy and that is sort of like that's why we're way side it's not a big priority

well nuclear nuclear is coming online suit like that I'm a huge fan of that I don't know how the

hell we got in our heads that like nuclear is is bad especially given the politics yeah yeah exactly

I mean it's yeah it's starting to actually I don't know how far along that is we've covered

many times sort of many nuclear and then sort of promising looking technology but is it getting

to a point where it could be powering data centers yes there are a bunch of data centers that

are like have plans for nuclear buildouts SMRs are probably going to be the biggest kind of sort of

shift expect the first SM like these are small modular reactors you can expect the first to come

online sometime like the most optimistic I've heard people like who are actually like in the data

center building space say is like 2031 maybe 2030 so it's like you know if you believe in Leopold

and whatever AI 2027 like we're away into super intelligence territory by the time that matters

but who knows yeah and you know how much this super technology has helped you with stuff

building nuclear safely anyways onto projects and open source and we do have a couple of

interesting new models that are open source starting with Neemotron dash labs dash diffusion

a tri mode language model unifying out of regressive diffusion and self speculation decoding

this is from Nvidia in their Neemotron model of family of models and the short version per

the title here is so you can do decoding you can do language models in different ways right the

standard way is out of regressive meaning that you output one talking at time effectively so you

provide the input you give the output and you keep sort of doing the solution diffusion as we've

discussed many times is something typically done with images where you sort of like produce the

entire thing at once over multiple steps and sort of make it better and better and you could do

that with text interestingly so instead of going left to right to word by word you kind of produce

the entire paragraph at once and denoise it until it's correct and then there is self speculation

decoding which is kind of like standard outer regressive but you have within the model some

shortcuts more or less that let you be quicker with diffusion also can can potentially be quicker

so in this tri mode approach they trained a single model that can do any of these and also has

been trained on each of these objectives and it allows you to switch between causal and

bidirectional attention meaning that you can use these to training objectives and they releasing

here base instruct and vision language variance at 3 billion 8 billion and 14 billion parameter

models the models are very quick so six x more tokens per four past then for instance grand

free 8 billion while being comparable and I don't know how we got to this point where Nvidia has

produced the most interesting research on neural net architectures and potential alternatives to

transformers I guess we've seen from academia mom and so on but as far as like large scale AI model

demonstrations of something that could be better than transformers I mean this is pretty exciting

yeah I mean coming from my guess a company that knows the hardware so deeply you know if you're

hypothesis is that progress is hardware is determined by hardware this may feel like a natural

place but we just hadn't see it to your point like you know the last time when we started covering

the big Nvidia open source releases I think you had to go back years to nematron

turning in LG the kind of big Microsoft video collaboration from my I want to say 2021 or

something was like the latest you know the the biggest case that you could point to and now suddenly

here they are this is a really interesting paradigm I mean as you say it's this combination of this

you think of it as like auto regressive modeling left to right and the diffusion piece and they do

a couple things kind of make it all work together conceptually so the first is they do still have to

use just standard traditional auto aggressive pre training to get the model up to speed right to set

its priors and then they turn on this dual track auto complete end diffusion thing but you see

that a lot like there's often a need to just use auto aggression to get these things off the ground

once they turn on though this dual track thing they do it in blocks and so instead of having

your entire output window the way diffusion works is you kind of start with random noise for all

the tokens in the output field and then you were kind of gradually populate them and get more

more confident as you do more steps of diffusion and you zero in on what the text needs to look like

and of course to get the benefit of being able to look at the whole body of output text at the same

time whereas with auto regressive modeling the challenge is once you choose a token now you're

stuck with that token for the next one and so if you go off in the wrong direction you can't go in

like 20 tokens later be like ah that's no good and so you'll see the model is kind of doubling

back a lot as a result of that because that's the only way they can get themselves back on track

and so what they do here is they're going to do what they call block wise diffusion so you split

the sequence into blocks and the model denoises one block at a time it kind of treats the earlier

blocks as clean context and so it's by directional it looks in other words at the whole chunk of text

within a given block but it does lock in past blocks as it goes so it's causal it sometimes

referred to it's kind of like a mix of it too basically and this is something you've seen already

as a potential pattern but this is to my knowledge we like biggest demonstration of this idea

and in general like this hybrid approach yeah and and they also they even have this like interesting

so the backbone of the transformers the residual stream it's kind of like the the the the vector that

gradually gets modified at each layer as as more layers add sort of more I'll call it context but

you know more more information to the vector and let it evolve and what they have is what they

called dual stream attention so they have this like this clean stream that's just causal it's

just used for the autoregressive part of the modeling and then they have a noisy stream which they

use for the diffusion loss and they're they're processed together in the same forward pass and so

both of those objectives the diffusion objective and the autoregressive objective they're computed

at the same time and so that's how you kind of get these to work together they find that there's

just like number that they use to vary the balance of how much autoregressive to how much diffusion

to put in the objectives which is a very common thing but the interesting thing is that they find

that the losses peak at the same they kind of like rise and fall together both the diffusion and the

autoregressive losses which is not intuitive and very interesting it suggests that they're complementary

rather than fighting for model capacity in some sense kind of interesting across the board and

conceptually I think one of the deeper attempts that I've seen to marry these two concepts together

and one the more open source release this one a bit less to say but still interesting

10 cent has released high free and opened 295 billion make sure of experts model with only 21

billion active parameters so the short version is this is not as good as GLM 5.2 but pretty decent

pretty decent intelligence and it's smaller and it's just another model that's like pretty solid

it's completely open source it's pretty cheap you can use it potentially even in a home compute

cluster so the models for now are continuing to be released but they may or may not keep

doing that on to policy and safety and we begin with some interpability research effectively

from on froupic and potentially some kind of safety philosophy whatever you want to call it

and this is I think the second most exciting news of a week in addition to the model releases so

we'll probably spend a small amount of time getting into the details so the name of a research

for on froupic is verbalizable representations from a global workspace in language models and I'll

try to provide the crew TLDR and see if I can manage it so they begin on froupic with kind of

laying the groundwork conceptually of what work space is and this is a model that has come out of

modeling consciousness in humans where one aspect of it that you can pinpoint to is consciousness

effectively is attention mirroring attention kind of directing it and you have this sort of workspace

where you slot in some stuff to be consciously aware of most of a processing of information

sensor data etc is unconscious you're not aware of it but it's happening and then you kind of

bring in bits to be looking at and thinking about like the active thinking you're doing the

active verbalization of things and things you're keeping in mind to draw on kind of that type of

your consciousness is a this workspace notion where a bunch of stuff is happening but you're not

conscious of and then there's this thing that you can kind of use consciously and that's effectively

what froupic is at least comparing this method to so the gist of the method is this kind of idea of

via some cool math that relates to Jacobians and whatever else you want to get into

the gist is you can find the tokens that the model seems to be kind of potentially ready to say

that are on its mind so to speak so this is very different from the traditional interpability technique

we've talked about a lot where you can look at vectors within a model that you can try to map to

concepts the difference here is instead of looking at vectors that sort of roughly map onto concepts

which is sort of this like unconscious diffuse messy kind of soup of stuff going on inside your

model this is telling you for a given token or given token sequence this is sort of what explicitly

in terms of tokens in terms of words is going on inside a model and there's a whole lot to say

about the technical aspects of this the individual ability aspects if you will not be able to get into

all of it but I'm sure Jeremy you have a bunch more to say yeah everybody be prepared to roll your

eyes for the annoying the classic things that you see so you know imagine every we talked about

the residual stream being the backbone of the transformer right this like vector that keeps getting

updated for a given token every layer until finally you get to the last layer and at the last

layer that vector gets decoded into a probability assignment for every possible next token so you

have this vector this list of numbers that in some sense encodes the meaning of the vector you're

about to predict but then you need to multiply that vector by a matrix a decoding matrix that turns

that into concrete probabilities for the word the the word apple and so on and so historically what

people would do to try to answer the question you just raised right what is the model thinking at

some middle layer which is kind of where you want to if you want to monitor the model for suspicious

sketchy planning if it's going to kidnap your children and murder you in a dark forest you want to

look and be able to tell is it thinking those thoughts somewhere intermediate in its intermediate layer

basically what is it not telling you right yeah inside of it aside from the output what is going on

exactly exactly and so one historical naive approach people would take because they would take

a decoding matrix at the end they would literally just like use it to decode immediately whatever

the residual stream what the residual vector was at any given layer so let's look at layer number

27 let's just whack it with the same decoding matrix that we would use on the very last layer the

problem here is that that decoding matrix was never optimized to work with that particular layer

it'll give you probability assignments over next tokens but it's not obvious what the hell those

are actually supposed to mean so somehow what you actually want is the ability to say okay if I

took this residual vector at layer number 27 and I modified that residual vector a little bit

what would the impact be downstream on the final predicted token let it prop it all the way through

those other layers and like if I just modify you know the residual stream layer 27 how then does

that change the output all the way up to layer 78 or however many layers there are when it actually

gets decoded and instead of whacking it with a matrix that was only optimized to work with like

layer 78 or whatever instead try to come up with some layer that approximates all of the shit

that happened between you and like that final layer that's what this is so that's what the

Jacobian here is doing it's an approximation and a pretty good one a pretty principled one of all

the crap all the transformations that happen between you know called layer 27 and the final layer

and the output and so it actually does give you in some sense the best estimate that we have is to

what the model was was thinking how it was let's say how it was trying to nudge the final out yeah

I think the the verbalization aspect of this because it is nuanced sort of you can look at the

embeddings you can look at the activations at layer you know whatever and sort of say what is

these activations telling us and that's the traditional way yeah so the the way this is different

from the traditional and to be really techniques is this verbalization aspect where you would you

look specifically at with token outputs the words the model would lead to as opposed to whatever

soup of stuff going on inside of it and at least we argument that anthropics making is that this

is akin to this workspace model and is actually interpretable as something akin to like what the

model is conscious of now wherever the true is a whole other topic with different perspectives but

it is a compelling and the kind of intriguing argument yeah well wait exactly the connection here

to the whole consciousness thing because it may not be obvious from our description so far like how

does this actually tie into that there's a couple piece of evidence so one if you ask the model

what it's thinking okay so if you if you look at the space of words that the model is thinking about

at layer 27 that you can you can figure out using this technique if you swap one sort of one vector

for another so so you know if it's thinking about soccer switch it for rugby that will actually

result in changes downstream that match exactly that so you can do these pretty surgical interventions

so so this means that there is like some some like causally connected thing here they also do stuff

like tell the model to not think about certain concepts and they find that actually these intermediate

layers are forced to think about the concept which is sort of funny it's like humans if I say don't

think of a pink elephant like first thing you're gonna do is think of a pink elephant but the key

thing is so the argument for for the consciousness thing is that there's a kind of reasoning that we

cannot access but there's also a kind of reasoning that where basically like all thoughts get promoted

to some shared reasoning space that the full human brain can access and that that kind of shared

workspace is what consciousness is about and this is essentially it allows them to sort of do a

a dissection of a language model and show that something similar is happening there where you actually

have layer like the earlier layers just like the model can't access the thoughts that are going

on there is just too early the the processing of the residual stream isn't yet to a point where

it's thinking coherent thoughts it's it's the thoughts are too raw but once you get those middle

layers you actually do get to that workspace and they're able to show through a bunch of experiments

that that this kind of maps on to intuitively how you might think about that in the context of

but you know the workspace associated with consciousness so quite interesting quite deep when it

comes to that I wish we had more time for this one but hopefully this gives a general sense yeah

honestly we could spend like two hours deep typing deep into this and maybe we need to

do the capability episode we keep saying we'll do one one day we'll do a deep dive episode again

but it's hard to find the time on the consciousness piece so I think it is important to note

nuance here where there's kind of a few things you might be referring to when you say consciousness

right there is sort of the functional aspect of consciousness which is like

their thoughts there's like a process going on this is how consciousness and thinking and

reasoning is working and global workspace theory is is part of that it's sort of trying to map out

okay here's how thinking happens right a completely different aspect of it is like

we experience of consciousness with kind of phenomenal aspect of like I feel that I'm conscious

and I'm aware of it and so on and so on and these two things are distinct we don't know why

you know we know our brain is doing stuff we know that thinking is happening in some sense reasoning

data processing why that data processing seems to result in our kind of conscious experience of

a world is the deepest hardest problem of philosophy the hard problem of consciousness and a

tropical is not making a statement as to that case so I think there's a lot of backlash against

and probably whenever they do this kind of stuff which I'm getting very tired of of like

unphropicators be like onphropic always mentioned safety always goes on about consciousness but

when you say consciousness and global workspace theory and so on what we are claiming here is

that functionally it looks similar and you can make a case whether it's kind of like thinking

workspace-esque mechanisms within the model which is plausible right because for various reasons at

least you can say we have this model of how thinking happens in humans there might be a similar

model you can apply to AI and this is an intriguing aspect whatever they should have framed it this way

in terms of research is a kind of thing you can argue about but yeah very interesting kind of new

tool in the interpability toolbox and I think another pretty powerful tool potentially if it turns

out to be reliable next Beijing is looking at curbing overseas access to China's top AI models

according to sources so Chinese authorities have held meetings over past month with top tech

firms including Alibaba by dense and see that AI but potentially restricting overseas access to

China's most advanced AI models including those not yet released this according to free

sources these are led by China's Ministry of Commerce all secrets but wouldn't be

surprising right if this were to come to pass China has nothing to gain basically from releasing

open source models to the rest of the world so we are getting some indications of it at least being

floated and the pieces being put in place yeah I mean it's you know we talked about this idea of

sleeper agents and I think that we'll we'll start to I mean if you start to see that Chinese the

the CCP actually endorsing the continued release of very powerful open source models that should

increase your baseline belief in the the probability that there's sleep rage and it's built in there

or that there is some kind of game of foot it doesn't mean that it's guaranteed it's just like just

think about the incentives and that's where you land they do want to have more market share

obviously on the open source side it is helpful in a whole bunch of ways but here the Chinese Communist

Party is clearly making that calculation themselves and going well you know we're not so sure that we

we like this so I will say you know for people who model the Chinese Communist Party as being

this benevolent the sort of like benefactor of the open source space and how wonderful it is that all

the open source models are Chinese yeah I guess hold your breath for the next two years because

you're gonna end up seeing a lot of similar moves just as everybody realizes the strategic

significance of these models and again it tells you something if they keep publishing this that tells

you only one thing I mean could tell you but it may tell you that something quite interesting

about what they're putting in those models so you know we don't know how the restrictions would work

there was a may round table of a bunch of Chinese legal experts who who produced them kind of

summary of another proceeding and in that case they were talking about a tiered system with like

basic open source models that would be subject to simple filings you'd have more advanced technology

that would face security reviews and then the most sensitive frontier models would just straight

up be barred from public release or restricted to domestic use so everyone seems to kind of be

converging on the same obvious truth which is you can't have weapons of mass destruction in the

form of like really really powerful AI cyber and bio models just in the hands of random people

and again it was foretold five years ago by a lot of people and a lot of people said it was

ridiculous at the time but like if you just kept plotting the scaling curves we were always going

to get here so this is in some ways the least surprising surprising news of the week right and I

think even the site from government interference deep seek is looking to IPO tensent and then buy

dense are massive so we might just see the same thing that happened in the US happened in China

right where at some point your models are good enough where it doesn't make sense to open source

anything you're competing with other Chinese companies and if you have the best model you keep

it to yourself and so either way I think this open source gravy train is probably going to end

as it has in the US and next up the X open AI employee behind AI 2027 recommends a Rosier

path so speaking of China this is about AI 2040 plan a which pervert title is from some of the same

team who did AI 2027 so quick recap AI 2027 is kind of a narrative that try to sketch out why you

should be afraid of misalignment and potential X risk and generally how things would go that could

lead to a catastrophic outcome or potentially a good outcome and it made a much of predictions in

the sort of narrative framework some of you should have been very accurate about for instance for

rise of coding agents government starting to play an active role in mid 2026 things like that and

AI 2027 it does predict takeoff scenario where at some point we get to course your self-improvement

something whatever you want to say and then the models get crazy crazy good everything is turned

upside down and AI can kill us all right so this is the follow up to what that kind of tries to lay

a narrative of a path where AI maybe cannot kill us all ideally and the gist of it

we catch folks and we're very high little plan is like okay so we want AI not to kill us all we

also expect AI to become super intelligent probably so what we need to do is make sure we have

alignment figured out before AI becomes like too hard to control and can kill us all and the way

they say we need to do that is slow down development right you need the models to not self-improve and

explode in intelligence super quickly we need to be like don't make us smarter until we figure

out alignment and then they can be smart and like solve all our problems and most of this AI 2040

thing is devoted to well okay let's say we want to do that how do you do that and in particular

how do you do that in both us and China right because you can do something in the US but then if

China keeps racing forward then US won't do that because you want to stay behind and so a lot of

it is spent kind of arguing or laying out a story and narrative of potential cooperation and

mechanisms to slow down AI development which have things like monitoring access to compute like

pretty strict direct government oversight of AI model development a bunch of responses to this

AI 2040 similar to AI 2027 again could go on for an hour and a half about it higher level you know

it is interesting to read about to see one sketch of how things how people are looking at the future

and if you are a believer in the fast takeoff scenario then it's probably nice to read about a

potential optimistic outcome probably an unrealistic like a way to see things in many ways but hey

I mean it's good to be optimistic and say maybe AI will not go so yeah and it absolutely like

and so full disclosure here my co-founder brother Ed and I both had a look at AI 2040 for it came out

and gave some feedback on it and stuff we have our our quibbles with it as everyone does I think

the top lines generally make sense I think you know this is a really thoughtful team that's that's

worked on this I mean Thomas Larson and Danil Coca-Tio in particular who like I've just known them

longer than the other folks who co-authored it but like very very thoughtful people they've been

right I mean like if you look at AI 2027 more than anybody else at the level of detail that they've

offered they've knocked it out of the park like there's like weird levels of correspondence that we

see between what's happening right now and what they predicted and it's weird seven is a while ago

I forget one but it was like early 2025 this should be exactly a while ago yeah something like that

where to the point where it's like you know people were we're we're laughing at it in ways that now

or just like oh I guess the US government is just like doing this okay yeah like wake up that's like

wake up and smell the scaling curves that's where they've been pointing the funny thing with scaling

curves is that something that compounds and like 10x is every year is something that like where yeah

the 10x will actually happen and like it doesn't do you much good to to believe it when the numbers

are small it's when the numbers are big that you're actually testing how good is your kind of empirical

extrapolation so anyway so it's it's age very gracefully what they do is they lay out a couple scenarios

we'll go into detail but like they've got this plan a scenario that they've presented in a lot of

depth here where they're talking about yeah the US and China get together and they're both really

scared about loss of control and other things so they actually set up this like international way of

coordinating around this stuff but they're they've got a plan B that's like China won't go willingly

and so we need to make them slow down plans a and b together or like exactly what we've been focused

on over the last like several months as we've been talking to like literally the diplomats who led

US China engagement on WMD risks to see like what is China like at the negotiating table table what's

realistic but also some of the hardware like compute assurance people who they've obviously spoken

to a lot of the same people they're proposing the same solution of network taps and recomputations

but trouble is when we take that to the intelligence community ask them about it it's it kind of looks

like a bit of a non-starter so there there's a bunch of things here where we're going to be coming

out with our our report pretty soon our differentiator is our ability to like actually talk to the

intelligence community talk to the diplomats talk to the people who do the builders of the data centers

and to take these kinds of scenarios just like we did with a situational awareness and be like okay

but how does this make contact with the actual reality of institutions and sort of special operations

and that sort of thing so we're going to be doing the same thing with this by accident so I'm not

going to kind of like give my full take on this but I think it's actually really really thoughtful it

does an excellent job and any quibbles that I might have with it are like within the range of like

things could go one way or the other and I don't think that they're meaningful so they got a bunch

of you know plan C and plan D things get increasingly shitty as like different governments don't take

it seriously enough early enough and then we end up in really bad positions it's worth at least

skimming especially if you are compelled by what AI 2027 said and how well its predictions of

age I would expect this to age similarly gracefully so check it out I highly recommend it more than

anything we've covered in the last sort of like three weeks or so I would I would recommend taking

a look at this yeah exactly I think if you look at the discourse as we've AI 2027 the mainstream

there's a very strong tendency of kind of the mainstream of AI commentators whatever you want to

call them AI community roughly to be at the best dismissive of these kinds of discourses around

ex-risk and around catastrophic outcomes around superintelligence a lot of skepticism in some cases

kind of constructive pushback yeah in some cases just like making fun of it and being like

vissa silly and in many ways it's been shown that kind of with this missile of these kinds of

safety concerns is is wrong like these are real yeah real things to consider I am someone who is

more on the like ex-risk skeptic side I think both AI 2027 and AI 2040 very heavily lean on

takeoff scenario of exponential self-improvement and that's the key question there's no question we get

to a GI but does a GI get us to SAI and does that get us to like models that are impossible to

control I think nothing in our current scale and curves or science really tells us much about fast

takeoff is my personal stance but you can make arguments either way anyway and that would be a good

episode for us to do at some point one of the things that I appreciate about the podcast is like

there is you do get like one take I think we're we're both like I may I may be more actually

outright like ex-risk piled but you certainly are more kind of a moderating influence in that

respect I think that's important because it's the reality has been rough around the edges like one of

the key things that people like me have been wrong about is if I had been correct you know five

years ago we'd all be dead by now I think that's very fair to say there are other aspects of the

story that that I was telling myself that sounded like science fiction of the time and like we're

exactly right right like the you know the mythos thing as an example but it's it's never as clean

as you want it to be in either direction and and and this is just we have to proceed accordingly

and it's not obvious what that means yeah the other thing that Silicon Valley is very bad at is

converting the technology progress into a societal impact and it's very easy to overestimate

the style so you know if you were I would have to go back and look at the at 2027 but part of these

projections is like insane changes in GDP and everyone being automated and every job being

automated and often it's a bit slower than you would imagine I think they're they're pretty much

on point weirdly on the the GDP and the workforce side the big numbers at this stage as I recall from

AI 2027 or more like a capEx spend that's the stuff that that sounded crazy and now it's just like

yeah you know of course we're we're spending like trillions of dollars on capEx that like why wouldn't

you that's fair that's fair but like GB beyond just like building more AI I guess yeah I can't

remember if they made this specific prediction but it's in the same ballpark it's like it basically

AI is the dominant driver of US GDP right now in the same way exponentials always look cute when

they're when they're young and then they become monsters very cheap GDP growth right in GDP growth

yeah yeah sorry sorry yes GDP good point but but that's you know that's I think in line I'd have to

go back and check but like Danielle posted recently I think he said something like by his estimate

were about you know 75% running at about 75% of the speed of AI 2027 and that matches my like

roughhand wavy sense of it but you know they're equipals at the margin everywhere yeah

and we're gonna have to close it out to there real lighting-based episode so much fun stuff to discuss

thank you so much for listening to this week's episode as usual caveat for me I'm sorry for

releases being a bit choppy and the timing it should be getting back on track as my startup gets

a bit less crazy we appreciate you listening sharing rating or podcast commenting will respond

to comments probably next week and more of anything please keep tuning in

yeah

I grew with the sheep and the future sees

Building up, building up your latest release

The class can pay I come into go-rides

Get the low down on tech and let it slide

The class can pay I come into go-rides

I'm a last-for-just-free say I's reaching high

From girl Netsu robot, the headlines pop

Made in driven dreams, they just don't stop

Every breakthrough, every code unwritten

On the edge of change, we're excited we're smitten

From machine learning marvels to coding kings

Features unfolding, see what it brings