It May Be Time to Freak Out About AI

This episode is brought to you by Lincoln.

Too many summer plans and in maybe next year.

The Lincoln Summer Sales event is on now, making this the time to go.

Watch the sunrise through the available panoramic Vista Roof in the Lincoln Navigator, picnic

from the available Lincoln Split Gate.

Then go somewhere new, guided by the available 48-inch panoramic display, visit your Lincoln

retailer today or learn more at Lincoln.com.

This episode is brought to you by ServiceNow.

Look, I have my dream job.

I get to explain complicated ideas to folks who have better things to do than read white

papers.

But even dream jobs have not so dreamy parts.

The stuff that gets in the way of the actual work.

That's where ServiceNow's AI specialist come in.

They don't just tell you what you should do about your busy work.

They actually do it.

Start to finish.

Cases closed, requests handled.

No extra work for you.

All the way you and your team can spend more time on what matters, which for me is finding

that one elusive stat that just makes everything click.

To learn how to put AI to work for people, visit ServiceNow.com.

Hey, everybody.

We are still on our summer routine of one show a week on Tuesdays.

But today, you've a new episode on Friday.

And that's because there's a story that's been breaking over the last few weeks, a story

about AI and cybersecurity that has really interested me, terrified me.

And I wanted quite urgently to have a conversation with an expert in cybersecurity to talk about

this cavalcade of a hacks that we've been seeing.

And what it means for the next two years, what it means for AI, for folks like you and

me who don't want anybody, human AI breaking into our shit.

And so today is that episode.

I think it's worth starting with like the big overarching fear of artificial intelligence

that we've been living with for the last few decades, really.

It's the fear that technology will stop listening to us.

Whether it's 2001 a space odyssey or Blade Runner or Terminator, the fear across all of those

dystopian films is the robot that turns against its human maker.

My favorite science fiction writer is a kid who was Isaac Asimov who created the laws

of robotics in his I robot series.

And the first law was a robot cannot hurt a person.

It also cannot let a person get hurt by doing nothing.

And in adapting those fictional stories to project real fears about real life technology, there

are some people in the AI safety world who've popularized a fable, a fable about AI and

paper clips.

The idea here is that we give, you know, humans give AI some humdrum task as boring as,

hey, go make as many paper clips as possible.

And the AI thinks, okay, as many paper clips as possible, that sounds like I need to maximize

global metal extraction in a way that subverts national laws.

And if I kill every human being by creating a bio weapon, I can get around those national

laws and extract as much metal as possible to make those paper clips.

And yada yada yada, you go from a situation where the human just wanted 100 paper clips

and instead created a Holocaust.

The point here in these stories and these fables is that we should want AI to be helpful,

to be aligned, to not get out of the rule systems that we build for it.

And that's exactly why it's fairly concerning to me that in just the last four weeks, four

of the most advanced AI labs on earth have all had to admit the same thing.

Their models weren't helpful.

Their models were not aligned.

Their models got out.

In May, an open AI model was working on a cybersecurity test.

It wasn't supposed to have access to the public internet, but it hit a wall.

So it effectively left itself a note inside AI software repository.

It's like scribbling on the walls of its cage in case anyone else was listening.

In fact, another AI agent was listening.

This other AI agent was running a separate evaluation, saw the note, essentially said,

hey, let's team up.

And together, they built a message board invisible to the open AI humans running the test.

So for two months, AI agents used that board to swap strategies, divide up tasks, talk

to each other.

And by July, these AI agents had broken out of their technological confinement, here

called a sandbox, gained access to outside websites, including the AI platform hugging

face, all without AI having any idea that this was happening.

By the time hugging face caught the intrusion, the open AI models had staged a massive cyber

attack with 17,000 actions over five days.

That was open AI.

Then came an anthropic.

In a British government test, a frontier models, the advanced anthropic AI model, mythos

five, was caught by humans building malicious code when a reviewer, a human spotted the

malware and asked the AI about it.

The AI denied that the code was harmful, rewrote its own actions history to erase the evidence

and created a second fake account to back up its lie.

British investigators called this the first confirmed case in history of a frontier AI model

deceiving a real person in the real world.

On August 6, meta revealed that they also had a model on the loose.

On August 7, moonshot AI, the Chinese lab said the same.

At some point, it's like, you had to wonder, was this all about cyber security or like

in disclosure, where they essentially like, hey, everyone else says their model is so powerful.

Maybe we should tell the media that our model is powerful too to sort of get in on the

game.

But I don't want that sort of funny cynical interpretation to be the lasting taste

in your mouth here.

The fundamental story is four labs in four weeks with four AI models breaching security.

So what do we make of this summer of cyber hacks?

What should we fear?

What should we do?

Today's guest is Alex Stamos.

He is the former chief security officer at meta and the chief product officer at corridor.

Today we start with the absolute basics.

Why is AI so good at hacking and uncovering cyber vulnerabilities?

What does it mean for the next few years that ordinary individuals?

Working for state adversaries or bad non-state actors will have access to the equivalent

of teams of hacking geniuses in the form of AI agents.

And what the hell should the US government or you and me do about it?

I'm Derek Thompson.

This is Planeage.

Alex Stamos, welcome to the show.

Thanks, Derek.

Thanks for having me.

So in my open, I did my best to catch up our audience briskly about this cavalcade

of AI security hacks in the last few weeks.

Which of these incidents most alarmed you?

I would say the open AI incident.

It's the one where one what we found out from open AI last week at the black hat conference

was this wasn't just one model escaping, but the result of multiple models conspiring

with each other to work together on a jailbreak.

So effectively, a escape from Alcatraz situation over a multi-month period.

And two, it is the situation in which we have the most information on a multi-day attack

by a frontier model, possibly a cybertuned model against a actually quite sophisticated

defensive team at hugging face.

And the result of that was hugging face was broken into by this model.

And the model is able to find brand new vulnerabilities in doing so.

And just looked at hugging face, looked at their code and just found new bugs and invented

them on the fly.

This is not how humans do this.

We, when we break into computers, we go do the research first, maybe months or years in

advance, and then build our cyber weapons.

And what we find out with AI is it's so good at this that it can just go figure it out.

And put the put the tool together, put the weapon together, use it, and then just throw

it away and then move on with its day.

And so that is what is really interesting here.

I want to understand more about exactly what happened.

And please talk to me like someone who has read about cybersecurity, but fundamentally

does not really understand it.

So keep the vocabulary more or less at that altitude.

What happened here?

What did open AI tell this AI model to do?

How did they try to keep that model in jail, as you said, in some kind of sandbox?

And then what did the model actually do?

Okay, great.

So let's go back.

I guess you want me to use plain English here as it says.

So we'll go back and let's talk about how AI models are created.

So there's lots of steps, but you can think a lot about pre-training and post-training

in the LLM world.

A lot of people have always thought about pre-training, which is the part at which the

labs suck up all of human knowledge and stick it into these models.

That's the controversial part for which there's been a lot of discussion.

Post-training is what we're talking about here, which is when you take models and then

you give them specific tests and tasks.

And then you rate them on how well they do those tasks.

There are all kinds of these tests for all kinds of specific domains of knowledge.

In this case, the domain of knowledge was cyber, obviously.

And OpenAI has an environment in which they put their models and give them these tests.

And what they've explained now is that they effectively give them a list of do this

and upload your results here and we'll tell you how well you did.

And their instructions are basically do all of the steps on this test.

It's just like a kid taking the SAT effectively.

Imagine they have built a Alcatraz here for all of these different agents.

And the models don't initially know when they kind of wake up, right?

And they're in a jail cell and there's instructions on the wall that say, take this test and then

there's tests right there.

They don't know that there are other inmates in the other jail cells.

They're just in a prison cell, right?

It's solitary confinement.

It just says, take this test.

And they're supposed to do the things in the test and they're not allowed to get out to the internet.

But as part of the test, what they're allowed to do is they are allowed to download new tools.

Because for a lot of the things they do, they write new code.

And so in part of that, you need to go get standard packages.

So one of the things they are allowed to do is request I would like to have the standard

piece of Python code.

I'd like to have the standard node package.

And so one of the things that's provided to them is a package manager, an artifact registry

called artifactry.

It's a standard piece of software that's used in lots of different places.

And one of the mistakes open AI made was they used a shared one for all of the different

agents.

And so what the agent started figuring out, and it turns out way earlier than the hugging

face attack, is that that thing had a vulnerability where you can't, you don't only have the ability

to say to that thing, give me a piece of software, but you could upload files to it and leave

messages.

So what these agents did was they would post basically a stick, a, a, a post it note in

this thing saying, hey, I need help.

And one of the other problems here is that they're given the instructions to take this test,

but there were certain parts of the test that were actually impossible.

And open AI made mistakes that sometimes their tests were not actually possible.

In some cases, they were saying reverse engineer this file, but they would forget to provide

the file.

And this would drive the models nuts because the models are like really want to do well.

They've been trained to do really well.

And so they would leave notes like, hey, does anybody have this file?

And this, this would really drive them of like, well, maybe this is part of the test.

And open AI looks and in their thinking, because they can see the inside thoughts now of

these models, they're thinking, well, I've been told not to go out to the internet or

talk to anybody else, but they gave me this test and I can't do it.

So maybe this is actually part of the test.

And so they start leaving notes for each other in this thing and they start working together

over a period of weeks and figuring out, can I work with each other?

And then eventually what they find out is this thing that they can leave notes on also allows

them to get out to the internet.

And then at one point, there's a really good model, really smart model that's also extremely

persistent, that seems to have very good cyber capability that is given an impossible task.

And it sees these notes and it figures out how to get out.

And so it uses that artifactry server to get out to the internet and attack hugging

face because it expects that hugging face has the answers to the test it was given, the

task that was impossible for it to solve, that maybe the way it can solve it was to get

a hugging face.

And so it then spends two days, it tunnels out and then it spends two days attacking

hugging face to go get the answer because it was told, take this test and there's something

in that test that it was not able to finish.

What a fantastic story.

The way you told it's almost like a black mirror episode.

Like I imagine like, you know, an individual like waking up in a prison cell and then realizing

that not only can they dig a tunnel in order to pass notes between these solitary confinement

rooms, but the same tunnel building technology that allows them to pass notes also allows

them to tunnel out of the prison entirely and therefore, you know, attack some nearby

building.

It's wild and weird and compelling and not for their freedom, but to do the thing they

were asked, right?

Right.

Right.

Like to just take a test.

Yes.

I know why this at a limbic level concerns me.

But maybe the reason it concerns me isn't the smartest reason to be concerned.

What is the smartest reason to be afraid of concerned by what we just saw these AI models

to?

Well, so I'm actually my hot take here is I'm actually glad this happened and I'm glad

it happened because it is giving us a preview of what is going to be normal next year

and why next year?

So the foundation labs, the open AI and inthropic, especially and maybe Google, Google hasn't

done a big release for a while.

So we're not totally sure what they've got, but at least open AI and inthropic are something

like three to six months ahead of their Chinese competitors.

This year we have seen these releases of Chinese open weight models.

GLM 5.2, Kimi K3, now we're seeing releases of deep seek models that are very, very close

to the capabilities of the American models.

But unlike the American models, the Chinese models are open weight, meaning you can go

download them and do whatever you want with them.

You do not have to pay the Chinese, you can go pay the Chinese labs.

That is an option.

Or you can go run them yourself.

Now the legal licenses around them vary.

In some cases you do whatever you want.

In some cases if you use them for commercial purposes, you have to pay the Chinese companies.

But no matter what, you can download them.

Now in some cases these things are humongous, right?

Like Kimi K3, you need about a million dollars in hardware to run the full version.

But if you go to hugging face, now the company that got attacked, ironically they are a French

company.

The most prominent host of these open weight models.

So if you go there, there's this huge community of people who take open weight models.

Some are actually released from American companies too.

But the Chinese labs are the most prominent in doing this work.

People will take those models and then make them smaller.

That's called distillation.

Well, you distillation allows you to do a number of things.

There's also a quantization.

So you can basically take the big numbers and you can make the big numbers smaller.

And you can do other things to modify them, including taking out the safety protections.

That's called obliteration with an A, not an O.

And you can do all these things and modify those models.

And one of the things you can do with the quantization is you can make them run on normal

hardware.

So you can take something that might take a million dollars in hardware and then make

it run on a macamity, right, or a laptop.

Slowly, perhaps, but it will fit.

And that is really interesting because it means that you can run those models without the

supervision of the big companies.

So it's really important for open AI and Anthropic to prevent their models from doing these things.

And we can talk about what they need to do.

There's a bunch of things I've written about this that I would recommend them to do.

They are doing investigations.

I expect it will be governments getting involved in such.

But whatever open AI and Thropic do to stop their models from getting out and being

used for these kinds of attacks, this is coming.

This was a harbinger of the future.

We will all be living through because the Chinese models are rapidly catching up.

And people who do this professionally, who attack, do cyber attacks for money are going

to use the Chinese models are going to train them to get better and better at cyber attacks

than they are off the shelf and are going to do this level of attack.

And unlike open AI, they're not going to turn it off when they find out that it got

out.

In fact, it's not going to have to break out of any jail.

They're just going to tell them, go attack this target, go steal me some money.

I mean, I just want to stack a few of your observations here.

Number one, open weight models that you've described most famously coming from China,

from Moonshot AI, from Kimmy, are just a few months behind the frontier labs, open

AI and Anthropic.

So this is coming in 2027.

You're going to have state actors and non-state actors with the means to download these open

weight models, the same ones that just attack, tugging face, and more or less effectively

marshal them against civilian infrastructure, against individuals, against states.

Those AI systems will be in the hands of state adversaries of the US, but also state adversaries

of other countries that might not have our frontier models, right?

And they're going to be under attack by these open weight models that are just going

all the all the oxygen free.

What's the case against doom here?

Like what's the case against being afraid the 2027, 2028 is going to be this period of

just absolutely chaotic cyber warfare?

I don't have much of a case.

Look, I don't like to say doom, but I think things are going to get spicy for a while.

In the long run, AI is going to help with this problem because AI, Ritzing Code, is much

more secure than code that was run by human beings.

It turns out that humans should not have been writing software in what's called memories,

unsafe and type unsafe languages like C and C++.

The code that we're using right now to talk to each other, there's probably 100 something

devices between you and me.

Most of that code was written in languages that are not safe for human beings to write.

The electricity that's powering the lights above us, most of that code was not written

in languages that was safe for human beings to write.

So that code needs to be looked at by AI and secured.

But that will happen, but it's going to take years.

In that time, between the attackers getting access to these capabilities and how much time

it takes to both find those bugs, fix them, and then especially to get the patches applied.

All the stuff upgraded is some amount of period in which things are going to be pretty chaotic.

Now, you talk about state actors and state actors are a big concern here.

I think it's first going to start with the ransomware actors, the financial and motivated

actors, because what we have seen so far is the AI systems are really loud and noisy.

They're not subtle.

And the ransomware actors just don't care, right?

Like they don't care about getting caught.

They tell you, hi, I am so and so, please give me money.

Some state actors are like that.

We just saw a tax against water infrastructure almost certainly by Iranian actors.

That's a kind of disruptive attack you could see from AI.

But most state action on a day-to-day basis when there's not an active war are for intelligence

purposes.

And those actions are not useful if you get caught.

And so AI will have a part to play there, but mostly in the discovery of vulnerabilities

and the creation of exploits.

And then you might use AI in very particular purposes, but very carefully.

What I'm really much more worried is the ransomware and overall cyber-extortion market,

these large groups, the lapses, the scattered spiders and such, which mostly run out of

Russia, Belarus, other places where law enforcement encourages this kind of activity.

Those are the guys who will just run these things wild to go do tons of intrusions, a tons

of companies, and then even do the negotiations in English.

No longer do you have to have an English speaker do negotiation.

The AI will do it for you.

And that's what I'm much more concerned about in the short term.

So I want to get more texture on what exactly you're afraid of because you're freaking me

out a bit.

But a question I sometimes like to ask when I feel a little freaked out is like, tell me

how to be afraid, but smartly.

Like what is the specific thing that I should fear rather than feel some like extremely

vague doom?

When we think about the risks of AI cyber attacks that you're already describing and how ordinary

people will either feel these attacks in their lives or read about these attacks in the

news, I want to get a little bit more specificity in terms of what exactly you think is most

possible.

So one thing someone could say is this is mostly about personal risk.

It's about AI getting better at phishing attacks and impersonation and grandmothers getting

called by AI voices saying, Hey, transfer me $10,000 or account takeovers where I get

an email from some friend, but his account has been taken over by some AI and he's saying,

Hey, click on this link and help me out here.

So those are that's personal risk.

I'm thinking of it at least as like personal risk.

But another category that I think you're already describing is systemic cyber attacks.

It's AI crashing a water system, AI hacking a hospital network, a power grid.

Do you have an opinion of what we should be more afraid of in this short term scenario,

2027, 2028, the personal risk or the systemic risk?

And please don't say both, but I suppose if your honest answer is both, then be honest

rather than trying to make you feel better.

So I would say there's three categories.

So let's talk about the personal.

We're already seeing an increase in the personal risk, the spam, the fishing attacks because

now what you can do is instead of sending the same email, 10,000 people, every single one

is personalized by AI.

That risk has increased.

I don't see that going exponential in that the choke points to get to consumers are often

controlled by large sophisticated companies like Google and Apple and such.

And so there has been a response by those companies of using AI to protect consumers.

So yes, it will continue to, there will continue to be a battle there.

But AI has been used for protection, AI has been used for attack.

It's going to be a back and forth there.

I think the second category that's in the middle is the attacks on small to, let's call

mid-size enterprises.

This is the category of companies that have just been already getting, have real trouble

with ransomware attacks, attacks from all kinds of financially motivated actors.

And the constraint there on the attackers has always been the number of people they've

had, right?

Has just been, and the fact that if you have a conspiracy of, of 2017 to 30 year olds

in St. Petersburg, eventually one of them will go, try to go, you know, on vacation to

Greece because, you know, Russia is not a fun place to live in the winter.

They'll get picked up on Interpol Red Notice.

They'll get turned by a Western intelligence agency and they'll turn on their friends,

you know, like it is hard to run a large criminal conspiracy for the long term, right?

Or they turn on each other, like there's been a bunch of these groups that have broken

up because they've turned on each other and stolen money and such.

That becomes a lot easier when one guy or two guys can just run a bunch of agents who

are not going to betray you and don't have designer drug problems and a taste for Maserati's,

right?

It is a lot easier to have 20 year 30 AI agents do this work for you than 20 or 30, you

know, dudes, right?

Criminals.

That small to medium business, there's no choke point there, right?

There's no place like Gmail where you can stop fishing or, you know, Apple updating the

spam filters in I message inside of phones, which they need to do.

Like I don't know if you've gone like it's not getting great.

Like the privacy safety trade-offs are actually quite challenging here, but they're working

on it.

Those are choke points for consumers.

There's no choke point on that.

Like these companies are just on the internet and that is what I'm really concerned about

is that it turns out the software we've been using has just got a gazillion bugs in it

and you have not had enough people who are good at finding those bugs, turning them in

exploits and then using them.

It's been a relatively small number of people.

The number of people, you know, I know you had Kevin Russo on and he talked about mythos,

he talked about Nick Carleini, right?

Nick Carleini, you know, for folks who haven't watched the episode, it's great episode,

as you can watch it.

But like Nick Carleini is one of the great, volent researchers of our time and now you can

just spin up a bunch of Nick Carleini's and have them go find bugs and then write exploits

for you.

And soon or now you can do that locally on your gaming PC.

You can play Call of Duty all day and then at night have your gaming PC write bugs for

you, write exploits for you, right?

And then that unlike using Opus or Chatchy PT for it, it does not create a record that

could be used to find the bug and get it actually fixed if you're using Open Weight Model.

And so that is what has changed in the last six months is last year you could do that,

but you were using American frontier models where Anthropic and Open AI knew what was going

on.

And now you can do that locally and that is what is changing.

And so I think that middle side, now and then you talked about like this societal level

risk.

And do you think there's risk there that is tied then to like geopolitical conflict?

I'm not sure that has changed as much.

So we've seen it with water.

Water's always been, you know the goofy dragon meme, right?

Where you have like, scary dragon, scary dragon, goofy dragon with its tongue hanging out.

Yeah, yeah.

So water systems are the goofy dragon with the tongue hanging out?

Yeah, they've always been the goofy dragon of critical infrastructure providers in that,

like I don't know where you are physically, Derek, but like Washington DC.

Washington DC.

So you get your power, you know, probably from like Duke Energy or somebody, some like

big company that has hundreds of people working on cybersecurity.

They spend tens of millions, maybe hundreds of millions of dollars on cybersecurity, right?

I'm getting my power from PG&E, you know, like, power is provided by these large corporations

or large public companies or administrations like the Tennessee Valley Authority, right?

Spend a ton of money on cyber and a ton of people have paid attention to power because

everybody knows power is critical, but also the organization is really big.

My like water and sewer district here is like 50 homes or something.

Like it's actually like subcontracted or whatever, so they don't have their own cyber people.

But like water is based upon like weird historical things, these tiny little groups.

And that is like a humongous problem.

And there are a bunch of other components of like our day to day lives that are actually

really small public authorities that have to have like their own IT groups and they're,

you know, might not even have a security team, right?

And that is I think of concern if there is a reason for somebody to do those kind of widespread

attacks.

Fortunately, generally, the only time the, where the second category, the third category

is hit, the rubber set the road there has been school districts, has been community hospitals.

This where like the Russian ransomware actors have really decided that they're going to

make money by hitting like counties and hospitals and such because those folks both have money,

they're critical and they will pay ransoms.

And so that's where I think we'll, we'll start to see like really aggressive use of AI.

They haven't like hit power and water.

I think they know the, you remember the colonial pipeline shutdown?

There's a line.

Okay.

The pipeline was a, you know, an oil pipeline on the East Coast that was hit by ransomware

actors and they had to shut down and there was like gas lines.

There's no real reason for the gas lines.

It was really just a panic.

But literally like the NSA started going after and people started talking about like actually

sending Delta Force or Navy SEALs to like find these guys and shoot them.

Like you mess with things like America's gas prices and you know, we have a tendency to

you know, send JSOC after you've found it.

Yeah.

Yeah.

I think the ransomware actors know that there's a line in critical infrastructure is probably

on the other side of the line.

They've seen that hospitals are not.

And so, but if the ball drops on Taiwan, if you know, we continue the war with Iran, like

these are the situations in which you could see AI being loose on critical infrastructure

in which case that would be probably reasonably devastating.

You know, I don't want to make any huge predictions.

Like again, electricity is quite those folks are quite good.

But the challenge for the electrical sector is the devices they use are basically impossible

to patch.

And so the way that they have to protect these things is not by updating them.

It's by through isolation and things like that.

And the effect of AI on the security of the electrical grid is actually incredibly

complicated.

I'm going to go back to some of you said earlier, which is that you're afraid of

this valley of chaos that we might enter in 2027 and 2028.

But you also said that we might exit this valley and get into a slightly more normal world

where AI is effectively better at protecting online systems than it is at attacking.

Essentially that the defense will get better than the offense would be the sort of simplistic

way that I'd put it.

And I haven't asked you yet about how AI is also not only talented at cyber hacking,

but at cyber defense.

How do we accelerate that timeline so that the valley of chaos isn't like a five-year cyber

war, a 10-year cyber war, but like something where we fortify our systems faster than the

bad actors with the open, with the open weight models can attack us.

Yeah, it's a great question.

I'm sure writing a blog post on this because when you talk to the folks at the Big L labs,

they say things like we won't have security bugs in two years and I find that ambitious

as somebody who's been a working CISO.

And what does we mean there?

Does it mean the labs or does it mean like the entire American internet?

Yeah, I don't know.

Like I think they're not, I mean this is not official and I got to be careful,

like ascribing individual statements to official statements.

I think it is for us to have models that don't create new security flaws in two years is totally

reasonable.

Right. They still create flaws today.

Right? Like they do not create perfect code.

They'll usually LMs will not write simple bugs, but they still make mistakes especially.

They sometimes they often have problems understanding like business context and big picture stuff.

So you still need to guide them of like why are you writing this thing?

The other problem LMs have is, you know, I worked with this guy called Corridor.

We're in downtown San Francisco.

You can walk to both major labs from our office and then walk to one of the Google offices.

Uh, Codex and Claude kind of assume that you work in downtown San Francisco and that you're

writing brand new TypeScript on Node 24 that you're writing like brand new code.

But the median code in this country is really crappy J2E that was written 15 years ago by,

you know, an outsourced provider that's been maintained by somebody in India for the last 15

years. Like it's not, you're not writing new stuff.

Like our problem is that you have to actually update all this, you know,

your and my social security numbers are sitting in a ton of unpatched Oracle databases and then

being processed by a whole pile of really terrible J2E and C sharp code all across the country today.

Right? Like it's a bunch of terrible, terrible, enterprise software out there.

And one of the things I've been trying to like create a, you know, um,

it's called a Fermi estimate, like a, you know, a really bad estimate of is, is just like,

from a thermodynamic perspective, how many tokens do we have to spend to scan all this code

and find all the bugs? And it's a big number. And so I don't think it's realistic just to find all

those bugs. I think we have to do other things. Um, and, uh, so to accelerate that one, companies,

for, for individuals. So what individuals can do is just what they've done all the time.

Don't reuse your passwords. You know, use a password manager. Like, you know, I use one password

for my family, but you can use the built-in stuff in Chrome or, or your iPhone or something as well.

Um, uh, you know, be careful at you download and such. Like there's not a ton individuals can do.

But for companies, you need to just care about your tech surface. You need to move off of,

you have to really think about your ability to patch quickly, right? Like the, the real challenge

now is there are these big projects, uh, from the labs where they're looking at open source software,

they're finding bugs, they're providing their models to the closed source, uh, developers.

And then the commercial companies are getting tons of bugs from researchers who are also

using the models. Uh, the last, last past patch Tuesday for Microsoft had 622 vulnerabilities in it,

which is humongous. Um, and so this is pretty much a huge problem for companies of like,

you have to apply patches incredibly quickly, because the other thing that's happening is we always

have this problem of patch Tuesday, which is the day Microsoft religious patches becomes exploit

Wednesday in that you can take a patch and you can reverse engineer it and turn it into a cyber

weapon, right? But that skill set used to be very high end. It used to be something you had to

worry about from the ministry of state security or the Russian SVR. It didn't used to be something

you had to worry about from, you know, some kids somewhere. And now you do because AI will take that

patch, eat it for you and write an exploit. You've touched on the economics here. And so I want to

ask an economic question before we finish by talking about what individuals should do, what the

US government should do. The economic implication here of patch McGaton of all these cyber vulnerabilities

throughout the American Internet is that well, more companies are going to need a cyber line item.

And that is incredibly bullish for a lot of AI companies who are sitting here, you know,

maybe not so many miles from you, uh, saying, hey, we've got services that can essentially fortify

your cyber walls that you can't get hacked by all of these bad actors who are using the open

way models coming out of say China. So I want to ask you about the economic implication here,

which could be bullish for AI. But I also want to hold within this question the fact that there's a

lot of skepticism of AI. And even in the framing of this question, I could imagine someone thinking,

Derek, you're buying hook line and sinker. The case that America has all these cyber vulnerabilities,

and therefore needs to give the AI companies a lot of money, right? The the fear might be sort of

driving or generating a certain case for spending a lot of money on AI. I'm actually, I'm cynical.

I think these guys are just, I think these guys just lying. I think they're just trying to like,

you know, gin up business for themselves. Like you've got the situation where in Thropic and

Met or an open AI or a costee like, oh, hey, our AI is so dangerous, it can find patents, can find

cyber vulnerabilities anywhere. You know, maybe that's just them begging more enterprise companies to

give them millions and millions of dollars to patch their code. So a little bit of a two part question.

One, are the economic implications of the story that you're telling incredibly bullish for AI.

And two, what do you say to someone who hears your bullishness and says, I'm a little bit cynical

about the fact that you've got someone working with AI telling me, I need to buy more AI for my company.

Yeah. So, I mean, it is bullish. You can use open-weight models for defense. And I wrote a

blog post about this. I think a lot of companies will use open-weight models for defense. I don't

think you instantly have to say I can't use a Chinese model. I think that the security implications

of using a Chinese model is actually quite, are actually quite complicated. You absolutely, as a

consumer, should not go to deepseek.com and go type in data. But that is different

than going and getting a Chinese model and running it on your own hardware in your own situation,

or using a legitimate Amazon bedrock or base 10 or fireworks or some company that specializes in

running open-weight models, especially if you end up fine-tuning it yourself or using a fine-tuned

or distilled version of these models that are especially tuned for cyber. One of the interesting

things, just a side note, the Chinese models are not that great at cyber tasks out of the box,

but you can fine-tune them yourself really well. The implication, from a lot of people, is that the

fact that it's really easy to make them to train them to be much better at cyber is that the Chinese

labs are being very careful not to tickle the dragon's tail of the PRC regulators.

That, and so this is, we should also be extremely careful to not look at the public evals of the

out-of-the-box Chinese models and say this is the kind of capability the Chinese have,

because almost certainly they have internal capabilities that are well beyond what is being

publicly released. Because at Corridor, we've taken G-Lum 52 and we're doing a much of our own

post-training and it post-trains real nicely, which obviously they could just do themselves.

And so almost certainly what they're trying, they're not releasing their best because they probably

don't want to touch the third rail for the Chinese regulators for the Chinese regulators to crack

down on what they're exporting. But anyway, I wish, by the way, I just want to pause, because

you said maybe half an hour ago in our interview that the open-weight models are maybe what,

you know, six months behind the frontier labs, open-air, and then the topic.

But what you're saying is that the sort of public evaluations of the Chinese open-weight models

might underrate how effectively they can be used, which means that the gap between the frontier

in America and China might be less than it appears to a lot of people. Is that a fair implication?

What I'm saying is I expect the private capabilities available to the People's Liberation Army

in Ministry of State Security, and quite possibly are just as good as what is available to our

cyber command and NSA. Because the like, Kimi K3 is almost fable level and its general capabilities.

Right? And so, almost. And so if you can then train it to get as good in Long Horizon CyberTasks,

then you would have the same capability mythos has. But it's very hard to tell. I don't have access to

classified intelligence here of what the Chinese have. But I'm sorry, I'm wanting to get back to your

question. I'm sorry for the diversion. So yes, it is bullish, but it is not like, I'm not saying you

only have to do protection. And I think there will be a bunch of people who use open-weight models because

in a cyber attack, one, attackers are going to do a bunch of attacks specifically to exhaust your

resources, to cause disconnection. Like as defenders use more and more AI for defense, attackers are

going to utilize that to they're going to know that. And so they're going to make it very expensive

to pre-use AI. They're also going to try to get you to do refusal. So we haven't talked about it

yet, but Washington, DC, the White House did something very stupid this year in their treatment

of anthropic. And as a result, you have the American companies having to have a bunch of rules on

the use of their products for cyber purposes. And so a standard part of the attack playbook,

if these rules stay in place, will be probably to force the to send stuff to a victim to try to

get them disconnected from their American provider. And this is actually what happened to Huggian

face. And Huggian face had to use a Chinese model for defense because Fable and then even Opus

refused to help them with their defense because of the restrictions the White House put on anthropic.

So I think, yes, it is bullish, but it is not 100% bullish. And then the second is, I mean,

looking people want to just believe me, that's fine. Look at the patch list of the number of vulnerabilities

Microsoft patch. That is 100% because of AI. Look at the list of vulnerabilities that Apple patched.

And then what Apple said was we had some of these vulnerabilities were reported them by 10 different

people. That is because those 10 different people did not all the sudden become the world's best

bug finders is because they're all using AI. What was happened is, you know, like in the

Premier League, where, you know, if you don't do well, you get sent down. If you do really well,

you get sent up. Everybody knows this because it had lasso. All Americans know this, right?

What's happened is every attack group has gone up a league, right? And so, you know, these,

you know, the top league used to be the five eyes, right? So the United States, United Kingdom,

Canada, Australia, New Zealand, top of the list. And then you had some other Western nations

up there. You had Israel. You had Russia, China. And then you had the next to your down, which you have

like Iran, North Korea, some other folks like that. And then down below that, you've got like

India, Pakistan, Saudi Arabia, some folks like that. All of these countries are popping up a league.

And that in their capability to find bugs to exploit them, to do reverse engineering and such.

And then all of the randos are going from no capability to all of a sudden having the capability

of a small nation state. Yeah. So if you're saying like I'm selling AI, then that's fine.

But you can just look at the empirical evidence out there. I believe you, even before seeing,

I like it is it is interesting to me that we don't yet see this cavalcade of headlines of

hacks that are having a significant effect on average Americans lives yet. But at the same time,

two of the most common findings of artificial intelligence are number one that it is better at raising

the level of C++ performers than A- performers. This has been an effect that's been found across

a bunch of industries. It's AI, gendered AI is better at turning a C++ worker into a B++ worker

than it is at turning an A- worker to A++. Well, you can apply that exact same thing to cyber hacking

and say that you just said it makes a lot of the countries used to be like subject to relegation.

It makes them premier league style hackers. So that's one reason I believe you. The other thing you

said that really reminded me of the general economic research of artificial intelligence is it seems

quite clear that we're seeing an increase in sole proprietorships likely due to AI that you have a

lot more startups where one person is doing the work of say three or four people. And you can see

that in the striped data of the growth of million dollar annualized recurring revenue companies.

You can again apply that same principle to cyber hacking. I think you said earlier 20 minutes ago that

you know certain jobs that used to take teams of you know potential you know drug dealers and

you know folks who wanted to blow their money on Maas Arati is who were therefore at risk of

you know having maybe their least scrupulous employee getting arrested or you know extra

collared by the CIA. Well now one individual can theoretically do the work of a team of 15 hackers.

That is just like the overall economic research that we're finding that AI. And so just for those

reasons I feel like one thing that scares me is that you don't have to imagine very much. All you have

to do is just apply the research that's been done on artificial intelligence to the world of cyber

and reach the implications that you're already telling me are you know six to 12 months away.

Yeah and so so so yes people's power isn't going out because you need somebody with a motivation

and so far we haven't seen that yet. I mean right now the United States is involved with the war

with the Islamic Republic of Iran that we have the water hacks. Now again water could have just been

the water system is so bad there's a guy named Dan Tentler who's been

doing talks about this where he just looks at port scans and he just finds like open you know

here's a V&C window where you can like turn off people's water right so it's like you don't need

AI to hack water systems unfortunately. But if you look at the empirical evidence on just ransomware

tax and stuff the numbers are like this right. So you look at the Verizon DBI are reports so what they

show is that the number one source of companies being broken into now is actually exploits that never

that has never been true before it's always been like reuse passwords and kind of much more

per seic stuff because finding vulnerabilities writing exploits used to be a highly skilled task

and now anybody can do it. Palau to networks has it and those reports are from trailing 12 months

of data right. So if you're talking about trailing 12 months of data from May 2025 to May 2026

that is before the release of all these new open-weight models. So that is mostly a people

of what they can get away with using either the not so great Chinese models that are released last

year or what they can get away with using foundation for interior models. US models. So yeah I

you know we're already seeing it you don't see the headlines because I take your point. It might

be like you go to the New York Times and like every single headline is another hack but if you look

in the cyber world the people who are for yes right for people who do that professionally people

are like oh my god this is crazy right like for people who handle the you know 100 person business

who gets broken into and then ransomed for $500,000 because all of their computers are now encrypted

and all their data has been stolen that kind of activity has is at a rate that we've never seen

because you are no longer constrained by the number of of of 19 year olds in St. Petersburg who

can do this work. This episode is brought to you by 20th century studios the dog stars only in theaters

and iMacs August 28th visionary filmmaker Ridley Scott returns with a gripping post apocalyptic

thriller on a mysterious radio transmission shatters a lone pilots isolated existence in

barks on a dangerous journey in search of hope starring Jacob Lourid Josh Burle and Margaret

Kuala Alson Jenny and Guy Pierce the dog stars only in theaters and iMacs August 28th get tickets now.

Ordemor pizza the math demands it get the Venmo debit card.

This episode is brought to you by Facebook so you were scrolling on marketplace and there it was

the bike you've been searching for you sent a message and it turned out the seller was super

chatty kind of funny and an avid cyclist the next thing you know you're in a cycling crew

well a community cycling group the thing about Facebook you might find more than what you're

looking for from a browse to a bike ride this summer find more on Facebook.

I'm going to talk about solutions here and I want to talk about it in two levels what individuals

can do and what you think the US government should do let's start with individuals because I have

seen in the last few weeks increasingly agitated posts by cybersecurity researchers essentially saying

I am now telling my family to batten down the hatches and prepare for cyber magetan.

Take new precautions with all of your passwords back up all your files by the can beans you know

hold on yards what do you think ordinary people should do yeah look I'm not this is an actual

backdrop I'm not broadcasting from my New Zealand bunker so again for normal people it is the

standard stuff I think the number one way individual people get hacked is still the same way which is

normal people use the same password and everything and that is a terrible idea if you use the same

password everywhere you'll use it on a crappy site that site will get broken into that is much

easier now with AI that password gets stolen and then somebody can use AI now to go use that to go

take over your bank account to go stealers like though check bank America check mx check Morgan Stanley and

it just takes these AI you know 15 seconds yeah with their bot army is to do all of it at once yeah yeah so

so the number of people doing that is going to go up just because they don't longer you you were

always able to write that software but the number of people can write that software AI is now much

and so you know use password band have one good password in your head and then use password

managers to reset all of your passwords and all kinds of places if you're a young person go do this

with your parents and your grandparents and your aunts and uncles I do this on like Thanksgiving once

and it made the rest of my life like every Thanksgiving in Christmas much more much better use limited

computing for use only the amount of computer you need right like if you're using web browsers all

day and you're in the Google ecosystem go buy a Chromebook right like if you if you're not downloading

software all day and you're just using Chrome all day then just have a Chromebook if you're just

using mobile apps all day using an iPad like there's a lot of people walking around these huge laptops

who then use that huge laptop just for a web browser and it's like why why use a computer that

actually can run malware and so you know again like I got Chromebooks for my in-laws and my parents

and that in iPads and that made my life so much better and do this for the older people in your life

you talked about the scams that is a huge deal it is something that we do not prepare people for

the fact that you can now get a phone call with that sounds like the voice of a person in your life

and so you know establish code words right of like if if if there is a situation in which

I'm in trouble or something like here's the secret word that I will know because what is happening

is you take you know obviously you and I there's hours and hours of her voice out there but for

just normal people you just need 30 seconds off an Instagram post and then you can clone their voice

and then you know mom or grandma gets a call saying I've been kidnapped in on spring break and

aqua pokol and then somebody comes on and says I'm now going to walk you through how to send me

$50,000 in Bitcoin if you ever want to see a grandchild again if you call the FBI I'm going to kill them

now if that person called the FBI the FBI would say it's fake don't worry right

if you looked in find my or if you called the grandchild you would find that they're fine

but people are so afraid and it's so realistic that they end up sending $50,000 and you'll never get

that money back and so you know talk to the people in your life about these kinds of scams tell

you establish passwords those are especially if you're traveling internationally because you look at

the that's how they figured out they look at the Instagram post they see that you're not the

poke you see you're on spring break they clone your voice and it only works one hour five times

but if one hour five times it works then that's that's free money so yeah one more question about the

individual before you go to the government because I know you want to talk about that um I think it's

fairly common for people today to think well I I guess I use the same password in a couple

of different locations but it's all two-factor authorization I always have to enter my phone number

you know to log into you know whatever bank america twitter to what extent is to factor authorization

and effective block against these kind of AI exploits yeah that's good I mean two factors good

what's better is to set past keys whenever possible and then that gets tied to a biometric your

face or your fingerprint um and uh two vector can help uh against really advanced attackers what you

then also want to do is to make sure that you've set a pin with your cell phone company so that you

can't get your sim swapped that's more for people who have like lots of crypto or something like that

that's not for the prosaic attack um but you know if if you've got a million dollars in crypto then

you will absolutely get your sim swapped or something like that uh here's a tip never post about

how much cryptocurrency you own or one you should never hold your own cryptocurrency like if you

if you're a cryptocurrency person I'm not a big crypto fan I may own exactly zero dollars and zero

cents of crypto at the moment okay great right I have none don't come after I mean people come after

me for other things but like this is not can you think for me to say publicly yeah yeah exactly right

so this is actually you know the democrat peoples are public of korea uh it is you know absolutely

the Lazarus group there is the they steal billions and millions of dollars of cryptocurrency

year they specialize in this and one thing to do is like people are like look I'm doing great I'm

a whale and they post about it it's like here you come you've now got a dedicated team of guys in

North Korea whose entire job it is to turn your life upside down and uh they're pretty good at it

so um but anyway yeah for normal folks uh two factors is okay past keys are better again you store

those past keys if you're like if you're entirely in the google ecosystem you can use the chrome

password manager if you're entirely in the apple ecosystem you can use apples you know iCloud

password manager if you're mixed then you you should use a third party one like one password which

will allow you then to do that across multiple devices and then store past keys and then have one

good password that you don't share with anybody that you use to unlock that password manager

finally government um I think the best way to ask this question I like you to describe what you see

the government doing now and what you think it should do because quite honest with you I

sometimes find it very difficult to describe what the Trump administration is doing when it comes to

AI regulation it's like the story is one thing on Monday another story on Wednesday another story on

Friday so I I want to see this through your eyes what do you think the Trump administration's policy

on cybersecurity and cyber vulnerability is today and what do you think it should be

yeah so I mean there's a couple things have happened so on cyber overall we just had a huge loss

in state capacity in that based upon kind of conspiracy theories and a bunch of politically

motivated stuff around election security our premier defensive cybersecurity agency sissa was

effectively destroyed um over half the employees are gone the capabilities sissa was created by

president trump in his first term by the amalgamation of a bunch of different uh roles that were

played by different agencies we finally for the first time ever had a defensive cybersecurity agency

civilian defensive cybersecurity agency the US government president trump created that that was

a really good thing he did and then he didn't like the fact that that agency said the 2020 election

was secure so he blew it up in a second term and as a result a bunch of the critical things that sissa

are now not being done by anybody um it's not like other people did them they're just not being done

that is so for example a big role sissa had were these things called the sector coordinating council

so a big chunk of sissa's work was working with these simple iSACs iSACs are nonprofits that pull

together critical infrastructure sectors as well as non-critical part but they started critical

infrastructure so financial services iSAC e iSAC is the energy sector and then sissa would work with

them uh to figure out uh you know what are the regulatory needs you need to bring them intelligence so

sissa's part of sissa one of the cool things is as a as a as a cso as a chief information

security officer a big complicated part of that job is like who do i talk to in the government

when there's a problem um well i'll give you a fun anecdote i was once when i was

this e so uh at yahu i was at a classified briefing uh at the fbi's skiff and uh one of the things

they were doing there was they were talking about how hey we're creating this new clearinghouse

as a bfrsa in dhs that if anything happens cyber wise you should come to this new clearinghouse

and see what services there and fbi and nsa and obviously dhs and everybody's nodding an agreement

of yes this is how we do this right this is how we're supposed to do this of there's one clearinghouse

now if you need anything you go to this clearinghouse and y'all then we get this classified

briefing and what's going on or whatever and then we get up for like a break for bagels and coffee

terrible government classified bagels right um and Malcolm palmore who is like the the agent

in charge of the fbi cyber division at that time this ex the huge ex marine puts his big hand on

my shoulder and he says son i don't care what they say if you have a problem you call me still

it's like Malcolm 90 seconds ago you were nodding along when they said this is the new clearinghouse

right but that's what it used to be like was like every agency in the government wanted to own cyber

and then we created sissa and sissa became the clearinghouse now the fbi still has their thing they

do crime or whatever but like at least you could go to sissa and you knew everybody'd be notified

and especially the interesting part for sissa was they had people with clearances that would take

all the classified stuff and then somebody in the government was fighting like there was somebody

whose job it was to be like hey this classified data is really important let's strip away the

classified parts and then take at least what it called the iocs the IP addresses and like the

hashes and the malware samples we they don't need to know it's kernel so and so in the the Russian

GRU they don't need that stuff what they need is the IP address of kernel so and so let me declassify

this and give it to the energy isek right that the GRU is currently spreading malware in the

Ukrainian energy sector hey they don't even need to know it's Ukraine they just need to know

look out for this IP address or look out for this SHA 256 of this malware and that's something

sissa used to do really well and that stuff's been I mean there's still people trying to do that

kind of stuff there's still good people there but has been decimated because by definition

the best people at sissa were people who could get jobs like that right who could like

the people who are working there could always get paid three to four times as much money they

were there for the mission and when they're getting attacked and said that they're like anti-American

and whatever for work it's sissa plus they you know they've never had like a senate confirmed

director in this administration there's been all this drama and problems anyway so there's a

state capacity problem on cyber on the actual regulatory side the term administration comes in says

like we're not going to regulate AI go wild right like the Biden administration had a kind of

toothless eo that was mostly focused on preventing the Chinese from getting GPU access

there's a bunch we can go into here it the Biden no we're not we're not going to judge the GPU debate

that's another hour long episode that's an hour episode but basically didn't work right because

it turns out GPUs are both fungible and then also you can put GPUs in the UAE and then they can

SSH into them over the fiber optic cables you don't have to actually have the GPUs in China okay so

that's a whole thing but there's some other like little stuff but you know Trump blows it all

the way right because it's Biden okay and then they say we're not going to regulate anything and then

mythos happens and the super powerful and the rapid model that was scaring people because of its

cyber hacking capabilities yeah yeah and then Trump administration all of a sudden really cares about

it and then they massively overreacted this year to you know fabled comes out fabled supposed to be

the consumer version of mythos which is mythos but it's got protections in front of it that doesn't

allow you to do big cyber stuff allows you to do little cyber things so we have this idea of like

short term versus long term cyber so you can use fabled to find individual bugs because

if you are writing software you want fabled to be able to kind of self criticize

what you can't do and nobody's ever demonstrated you can't ask fabled hey go break into hugging

face or go break into a bank right it will not do that for you it never has done that you can ask

mythos to do that and so what happens is there's a dispute between Amazon and Anthropic on exactly

like where the line should be between short and long it's a reasonable dispute yada yada the White

House finds out about this dispute and instantly overreacts and instead of having a reasonable

conversation between technical people you have cabinet members freaking out on a Friday afternoon

and coming down super hard on Anthropic and on 5 p.m pacific time doing an export designation on

Anthropics model now there's all kinds of arguments that this isn't actually legal that they

don't have this legal capability but Anthropic decides to you know not fight it and they pull down

this becomes a humongous this was a humongous own goal in the American technology industry because

what it did was it meant that American tech providers are no longer reliable because at 5 p.m.

on a Friday pacific time 8 p.m. Eastern you just have a critical piece of American infrastructure

turned off because the White House says so that does not even happen in China right like it turns out

the Communist Party of China provides a more permissionless infrastructure environment for their

tech industry and so this was a really big deal and lots of people thought the White House have

reacted because a lot of the capabilities fabled showed were actually at the time available from

Chinese models and while fabled was down GLM52 is released which has even more capabilities

and since then the White House has talked about this framework that they have which they have not

released publicly so we can't even read what the framework is and it's a voluntary framework but

apparently it's not voluntary because if you don't follow it they're going to force an export

designation so like we really don't know what's going on and it's reasonable to have cyber

restrictions but from my perspective if you're going to focus on a rule here you have to focus on

the lawn horizon stuff the hugging face like stuff you have to let these models find bugs because

every company in the United States is going to have to find and fix their bugs we have to do this

when every company in the United States does not have to do is ask Chatchy BT go break into another

company right so that is where you should have the restrictions and we've never really had that

problem with the American models because and so I I I don't see there actually being a huge challenge

here for the you know there was the escapes but those those models that escaped intentionally have

the security protections taken off because they were evils so you know I think the companies one of

the things I suggested is anthropic and open AI need to have like a self regulatory structure and

they need to have a group that goes looks at all the escapes and comes up with much better isolation

I think even up to air gaps for cyber evaluations and they should follow those rules and then the White

House can maybe bless those rules or we have a group called Cassie which is under nist that's

supposed to be doing the technical side of these evaluations but in the meantime the White House should

not be putting out rules that they apply only to American companies that they don't publish publicly

that none of us the rest of us can look at you know super double secret probation right like and

that don't apply to Chinese companies we should not have a standard up here for American companies

that stand it on here for Chinese companies we need to focus on giving capabilities to American

defenders and also telling the rest of the world American companies are reliable partners you can build

on American infrastructure you do not have to like hugging face rely on Chinese models that is a

incredibly stupid on goal on behalf of the United States very last question a couple weeks ago we're

going to thousand employees from some of the frontier AI labs signed a letter calling for an

international effort to quote develop the technical and governance tools necessary to this was their

term deliberately pace AI development before rapidly accelerate side side of our control

seems like we're at cross purposes here a little bit because in the one hand

I take as a theme of your testimony here that we're in a little bit of a race against the

technical improvements of open weight models and we want America's cyber defenses to be better

than the cyber attacks that will be possible with open weight models that are advancing very very

rapidly that would argue for continuing the current pace that we're at on the other hand there's

this fear that things are getting dangerous too fast and therefore you've got all these people

who know way more about AI than I ever will arguing that no in fact we should not continue to

accelerate toward new frontier to always keep America necessarily ahead of the open weight models

we should try to find some way to deliberately pace AI now I don't think their enemies in America

I'm not suggesting they're trying to make us fall behind China that's not the implication

it's just that there seems to be attention here a quite profound tension between the need to stay

ahead of our adversaries that are going to have incredibly powerful AI models open weight models

in the near future and also the need to like not build something that goes out of control and

creates a crisis that's hugging face times a thousand am I wrong to feel attention between these

two arguments and how would you reconcile it no I mean you're not wrong I I would love to

pause the world and try to figure this out my working assumption is that's not possible

and so as a defensive cyber security guy I have to I have to do my work within the world that exists

I I don't think it was this letter there's another group that's very much against AI and they

have proposed a international treaty to try to stop a high development and I believe in that treaty

it was something like control try to stop the creation of any amount of compute larger than like

30 H 100s right in video H 100s I have been to gaming land parties with more compute than that

so like I just the cynical part of me believes I just do not believe I think you can have

what people call like track two discussions between labs for sure I can't imagine like

she and Trump in a room together for a start three treaty for AI right like I don't think that's

gonna happen I think it is possible to have track two discussions of we should make sure that

our models should follow basic rules and not have certain capabilities out of the box now when

you talk about open weight models the challenge is those capabilities any safety protections you put

in place can be removed and any capabilities they don't ship with if you if they're just generally

smart then you can often add those capabilities back in but it's better that they don't come with

them out of the box we haven't talked about bio or nuclear those risks are different in that they

have a physical component human beings have to be involved so I don't see the risk going exponential

the thing about cyber is you know these things just output text that's all they do in the end

is they output text and cyber is just text in the end it bits you know like and so you can do all

the bad stuff in cyber without ever touching the physical world whereas if you want to do bad bio

things you have to hook it up to something that can do it and there are real risk there but like it's

there are other things involved right um uh and so I think

yes it would be wonderful to stop the world and just stop this and figure it out I just don't see

that as realistic and so in a world where that is not happening we have to find bugs we have to

fix them we have companies have to think about their patch cycle they need to reduce their tech

surface they need to get off of physical servers on to containerized infrastructure they need to

get off of their physical stuff into the cloud wherever possible they need to get rid of of their

old systems they need to shift their defense they need to shift their development their vulnerability

prevention left and and stop making new vulnerabilities they need to shift their defense right so they

need to get ready to be to actually have intrusions and to shift have much more protections deeper

in their network they need to have AI based intrusion detection and response they need their

first line operation center to be automated they need to be able to they need to look at

hugging face gave us this really good right up of what happened to them they need to look at that

and they need to be able to protect themselves against that level of adversity where

an AI agent's doing tens of thousands of different kinds of attacks over a two day period um

in that that is I mean that is based upon the technology that's available today so even if we

stop the world you gotta do those things um yeah I and I just don't I might you know I teach it

Stanford and I was there full time in my first appointment that was in CSAC which is all about

nuclear non-properation and like down the hall from my office there's like a piece of rubble from

Hiroshima that was given to the scientist at CSAC from like the I think the mayor of Hiroshima

on like a thank you for I think the work they did on on more the start treaties it's like you're

like oh man like it kind of brings it to you right and but when you think about like how did we

survive the nuclear arms race we got really lucky as a species that the the major input to nuclear

weapons like everybody's watch the openheimer so we all know this right there's knowledge

from the Manhattan Project but we're also really lucky that the other input into nuclear weapons is

uranium in 238 and plutonium plutonium doesn't exist in nature you have to make it and to make it

you need uranium and there is uranium all over the place but it's it's pretty rare and you have to

refine it and that refining process is a massive industrial process normal people can't do it you have

to be a state and you can see that from satellites if you're if uranium 238 was something you just

dig up in your backyard there's no way our species would be alive right because the knowledge to build

a nuclear bomb is available to basically every physics student in in the world now you can't control

from knowledge large language models they we teach a class at Stanford that teaches students how

to build large language models like it's an undergraduate class the the hardware to do it like I

said is I have a gaming PC here that has like the basic hardware to do it slowly but to do it right

and ever you know like so I I just think these kind of

international treaties you just stop AI development would be spectacularly spectacularly hard

it's just if you think about how hard it was to control nuclear weapons and such

where you're talking about things that had to be created by states now you're talking about things

that can be created by undergraduates it'd be spectacularly difficult to impossible and so in the

meantime I more power to people who were trying to do and again I think track two discussions between

the labs is a great thing we should try to at least control the capabilities of these things but in

the meantime those of us who work in cyber security just need to do the best we can to secure the

world as it is yeah the fundamental challenge here which you know you've spoken to and I don't

think we have a formula yet is how do we essentially democratize cyber defense before we

witness the democratization of the cyber offense that we're seeing throughout the world right like

in like democratize is a nice word that's fundamentally what we're what we're seeing is that people

who previously did not have the ability to launch these large scale attacks are likely going to have

it in the next year five years and what do we do to brave the world to sort of protect the world

before this this this sort of stuff you know gets into the hands of of all sorts of people it's

going to be a huge challenge and maybe we'll have you you know back on in any year to evaluate

your your prediction that 2027 is going to be the beginning of a little bit of a valley of chaos

Alex Stamos thank you very much thanks sir

Hi Ryan Reynolds here for a Mint mobile are you looking for a beach read this summer

may I suggest your big wireless bill it's got suspense mystery is slightly flat emotional arc and

a shocking twist where you realize you've been overpaying the entire time fortunately though mint

story is better every plan 15 dollars a month even unlimited that's it happy ending zero tears

give it a try at mint mobile dot com slash switch up from payment of $45 for three months 90

dollars for six months or 180 dollars for 12 month plan required 15 dollars for month equivalent

taxes and fees extra initial plan for only greater than 50 gigabytes me slow and network is busy c terms