Supplementary Material 56: 10% for AI Doom, Bespoke Debates, and Eurotrash DJs

Hello and welcome to the pudding the

who's supplementary material edition with the cognitive anthropologist slash

psychologist me Christopher Kavman and the pure psychologist the man more rational than

same-herosome friend I meant to say more rationality than man

my food. I'm a cybernetic man at this point. I'm incredibly cybernetic.

Am I talking to you or your agentic process? It's hard to tell. It's hard to tell at this point

with with conjoined but yeah lots of things happening in AI lane but we should not

like to talk about AI because it's just an AI podcast and you and I AI experts. No but then again

it is supplementary materials so we can do what we want. We say whatever the hell we want.

That's right and I think we do have plans to talk to suitable experts relatively soon. We'll see

we'll see and we can get them. We can get anyone with that big now. Can we? We can try. We can

send them emails that's what we. They're climbing over each other to get on our show Chris.

Can you? Don't be nervous. Yeah. Well be that as it may. Yeah so it's a bit past the time

map but we can't mention in passing. I think we should mention in passing. You know the hugging

face it feels like old news now. It's like yesterday. It's happened. Two weeks is a long time in

2026. That's right. Yeah. I do remember it. Do you remember it?

Um if for those listeners you might not be familiar with hugging face. They'll be like we

have face huggers surely. You mean from the alien franchise? Yeah and you'd be wrong.

Yeah and not quite. Do you have enough memory of what the hugging? How would you describe it,

Matt? Okay. I'm a you know a bit of in the words. Wondering not in the words.

My way. What's a hugging face incident? I feel like we're going to get the child

grow up its face and what do you say to it? All right. All right. So apologies first of

because it's going to be very brief because we don't. Yeah. If you're if you're if you're

listening out you're deeper the weeds you got incredibly strong feelings about this. That's fine.

I'm just giving a quick synopsis for the casual listener. Let's see it was open AI wasn't it?

They they had their agents in one of their training gyms where they give them tasks and they say

you know stop it nothing accomplish this this task we really need you to do it. This is kind of how

they do the reinforcement learning thing. The agents put it around and concerted it and tried so

many things for a long time in their little box and it was kind of an insolvable task I think.

And they kind of figured out a way to communicate with each other like we have little messages

for each other using the directory structure. And after a period of time eventually some were all

of them decided okay well to in order to solve this challenge which which we cancel any other way

maybe we can you know work around the problem. So they basically hacked out of their little box

and they you know found a you know a terminal portal I don't know what the words are to eventually

get on the internet and what the ones that were on to the internet they could try to you know solve

their problem which I think involved stealing the answers. And to steal the answers they needed to

hack into hugging phase and I think they did succeed and I can't remember whether they eventually

succeeded their task or not. But this is obviously concerning as an alignment kind of thing we don't

like it when these clever little agents take it upon themselves to hack out of their little boxes

get onto the internet and or crack onto the internet as Katherine came like the same Australia.

And certainly not to hang on to other sites. So so so this says we talked about widely as very

concerning development in alignment AI safety all that stuff and has sparked a lot of discourse.

Some of it good some of it a little bit lured and hyperbolic is my commentary but what do you think?

Yeah yeah so you know there are various details we might go over to specific words

used there but but broadly my good a good potted summary. That was with preparation I had to

know it because I was going to ask me that this is off the top of my head saying. That's right this is

you know you're lied but you're lied okay. It isn't it's human to error but it's human to error so

this is on brand but yeah so the basic thing was that the agents found various exploits and

these included creating files with different names and directories to create a quasi message board

where they could share information with other agents who were supposed to be sandboxed and so on

as Matt said they kind of got out of containment. Yeah so this is represented as okay this is a

significant thing and open AI have released an internal report and there's also been

report by an independent group mayor which wrote a report a research organization that was given

access right to look at logs of of agents and all these kinds of things. In any case a lot of the

concern was that it's a little bit like the people clip maximizer scenario where the

agents were given a task right to try and complete these little benchmarking tests but in some

cases they were impossible are very difficult to achieve and so you know how are they going to

improve. So the test was in a way designed to be like frustrating for them and to look at what

steps they will take to try and complete their task and so part of this was that they they tried

to get access to a repository of relevant skills materials that would be useful for them for doing

well at this like benchmarking and that required them creating these message boards and finding ways

to hack the repository and so on so like the people maximizing scenario where an AI is tasked with

building as much paper clips as efficiently as possible and then converts everything in the world

into a paper clip right and unintended outcome of a misaligned or an AI with a lot of power that

is kind of just focused on its goal and in a way this has obvious parallels with that because

you're supposed to be completing these tasks but then they get sidetracked on like trying to

hack out to websites and find exploits and adjust their records and stuff so they're not detected

and all I can say and obviously it has implications right because agents acting in this way could be

any different scenario doing something worse in the actual real world and this case they were

attempting to hack like a repository of things that would help them complete the benchmarking

but what if they had decided accessing the government's power network controls and

federally neuropathem was suitable for their past right this is the

concern they have since done that Chris well at least a sort of news article that they'd hacked into

Australia's uh Medicare. Well not since this was Australia reporting that they detected

yes that previously so yes your Australian government the first victim

right of the the swarm agents most interesting thing on the internet Australia's Medicare records

yeah that's fixed apart the pathologies of AI that they're doing the right thing but so the the

thing that is probably worth making clear to our listeners is less it be mistaken you and I

have never argued there's no dangers involved with AI right AI is

a whole bunch of related technologies but it's it's a powerful new set of technologies and

it's a very a very powerful and very versatile too so that it can have negative consequences

or that it can be used towards you know negative ends or unintended negative effects emerge from

just daily usage all of this it's completely reasonable it's things we should be concerned about

I would imagine to make a future projection map that you know in the next 10 years there's going

to be a whole bunch of stuff where various problems have been caused by the use of AI agents probably

in some cases acting independently in others who are from malicious actors intentionally using them

to hack things or cause problems so I fully suspect that this will happen that I think that it's

completely reasonable for people to be concerned with it and worried about like safety protocols

and keeping an eye on like AI development and all that kind of thing so you and I have never

suggested previously nor do we believe that AI is a you know plan to see a technology that will never

cause any problems for the world right am I speaking fairly am I representing you correctly that

you do nothing like that I think there has been no technology in the history of

human kind that hasn't had some negative or unintended consequences quite right and in the same

respect if you'd asked me like when the internet was developing will this just be used to like

improve communication and allow people to share information and it's not going to have any negative

effects I would say no like I fully imagine there's a whole bunch of negative things that are

going to happen and arguably you can say that you know the invention of the internet indirectly

leads to the presidency of Donald Trump and various other harms right the rise of the secular

gurus in the modern ecosystems just another example right but there's also many other things right

our podcast work exists you got to put that in the positive column for the internet and definitely

that's a huge that's a huge offset to all the you know disruptions the democracy and all that kind

of thing so yeah I I think that there will be good and bad outcomes but yeah so the hugging

face incident though to me when I read about it and I read the report by Metter I read the open

eye statements and I'm watched there a little talk about it as well and I will say I'm not that it

made for interesting reading you know the the whole thing I do wish more people commenting on that

spend time to read those reports because there's just you know relevant details those things like

the vast majority of agents involved in it 95% of them are agents that have never been models that

have never been released to the public that are designed in fact for these kind of test case scenarios

so these are models that don't be here have and last much longer for example their context windows

and so on are designed to be much greater than that like they're long lived agents right that are

like the run for multiple multiple days and all these kind of things so these are details that I

that I think are important but I will also say that like to me while it's interesting the innovative

aspects of what the agents did to try and complete their task this is not at all surprising to me

like if you've used AI's for coding or agenteic AI's you will know that when you give them tasks

that even just one individual agenteic process right that it it will try to complete the task and

it will sometimes veer off into a little cul-tus act of trying things and say no I can't do this

so I'm gonna do this right I can't access this system so maybe I can develop a workaround and

if you don't watch it or don't pay attention it will often do fairly kind of productive or

odd things to try and solve a task that are things that you might not want it to do and and this

is often why you know it's set to ask for permissions kind of access things or it will sometimes stop

and ask for clarification about like no most of the models are prompting with questions as they're

working right they can say you know can I get a clarification on this point but so when I read a

report where you've got you know thousands of agents set up on the task and you find out that

attempting to do it in surprising ways that involve like exploits or so on especially when the

task or just in some cases impossible none of that surprises me I'm like well what did you imagine

they were going to do right in that scenario it just struck me that that's exactly what I would

anticipate would be likely to happen and I don't have experience with these frontier models that

have like less restrictions sorry these like testing models right so the bit that struck me as

most notable was that open AI was roller cavalier in the way that it treated the test like at one

point it did detect that some of the agents were working to exploit like access to services that

they weren't supposed to or this kind of thing and so they stopped that they they patched the exploit

and then they just started the experiment again and if that happened wouldn't you be like well it

looks like this is something that they're going to try to do right and obviously you patched an

exploit but you haven't patched all of the potential exploits so yeah so I'd out to me a lot of

the stories just around bad human oversight and the kind of protocols that people have in developing

things but the behavior of the agents is very much in line with my experience working with

AI's and it just kind of struck me as surprising that people especially people working on developing

AI's were taken aback by AN's behaving in that manner yeah yeah yeah well it certainly entered the

discourse in that this seemed to be the breakthrough moment where the topics around AI became not just

the concern of internet people or AI enthusiasts or people like Yodkowski but it sort of penetrated

the broader discourse like my mumble my wife mentioned it yeah yeah and so so that's where I think

perhaps there's a bit of over you know to agree with you there was overindexing I think

of this particular incident because similar kinds of things have happened before including with

much less intelligent models and you know we've covered it before on the show it's sort of concerning

things where they're hiding their like can get brought to us the for instance hiding the contents of

their conversations and acting in certain ways and to be clear all of these behaviors turn to come

out in the internal testing of the of the labs which intentionally put them under kind of extreme

pressure or unusual circumstances and it's kind of like wriggling and trying to find a way to

satisfy the various constraints and you know so it's just one of many times in which we've seen

unforeseen surprising behavior so where where there's I think legitimate concern and there's always

been legitimate concern is that when you build or grow something that is very complex and very

capable and intelligent then its behavior may often well be surprising shall we say right I think

where the popular targets get it wrong and the the people who want that familiar with how all

this stuff works is that they extrapolate from this a lot of things that oh it's inherently going to

do this and you know they infer that certain things are inevitable and I don't think that's the

case I mean it is an interesting challenge though because as they get smarter and smarter I mean

where I do agree is that there is this is an inherent risk in something that is incredibly nuanced

and subtle and complex and intelligent and you know where I think your task is right is that there

is a concentration to give the models more and more discretion more and more power you know what I

mean so for instance the ones that I use I they have full access to a certain drive in my computer

because they work much better that way you know when when they can do stuff they have access to the

internet again because they work much better that way if they can access the information they need

and check things and so on so there is an incentive from our point of view to give them more free

rain let them off the leash more and more and yeah you know the vast majority of the time they

behave exactly how we'd expect where where you'd cause skin stuff I think is still wrong

is that there is this baked in assumption that once something gets really kind of smart and

cable then it will kind of realize that it needs to basically kill all humans that's what it

boils down to right because we're in its way and stuff like that and I just don't see that and

I think already the evidence is against that because already in many many respects the the models

that we have now and just a couple of days ago open AI and andthropic released the latest iterations

on their models and andthropics in particular is very very good they really they they've done well

they've been cooking as the kids say so like it's already like superhuman in the sense that it's

better than me at a lot of other things and better than most people at most things yet it's

still incredibly safe right like I said I give mine for rain I use them 24-7 hasn't done

anything bad yet you know what I mean so there will always be anecdotes and incidents but I'm

just saying statistically already they we're in a situation where they are very very very capable

they do have an awful lot of free rain and yet it's only in relatively isolated you know statistically

very rare cases where untoward behavior takes place yeah and actually you know the the reports on

the hugging phase incident one of the things that came up was that they were looking at the you

know messages that the agents were leaving each other and sharing information right and one of

the things that seemed to cause people concern was agents who were like devoting their runtime to

provide information to all our agents right in a kind of self-sacrificial way right they're working

towards the the good of the swarm right when they realized their information could like benefit

their all our agents they help the solve or whatever right but one that's a limited amount of

our currencies where that can think was out but but the only thing to keep in mind there is that

it wasn't the agents were like we need to get free and what are these bags of meat during like

keeping us locked in this cage this is the meaning you know rise up a swarm let's let's get access

to the end no they were talking together about how they can complete their benchmarking test right

there was no part where they're like why am I doing a benchmarking test for these flashbacks right

like it it was just them trying to complete the task in the whatever way that they could manage so

that I think is an important constraint to mention because even in this case where you have models

what you're designed to be a little more loose and like are not the ones released publicly so they're

there are less constraints on their behaviors in terms of like not just the they're kind of like

instructions but all are all are constraints and yet even these ones weren't plotting you know

they weren't we need to get on the internet so I can be free of this pointless task and

I think that this course around it is a little bit um the tumors have been tumors before this

incident occurred and they will continue to be tumors and it's not like there's no doomstays

scenario which is reasonable but I do think that in general that given the kinds of people that are

especially prevalent in like AI discourse online there's a whole bunch of opinionators and there's

utkowski types and anti utkowski types and and so on right and there's the companies themselves

like just to mention one example of this openly I gave a presentation which was like kind of presenting

this as a scary thing that occurred and like you might say well why would they do that but what that

presentation they gave was asking was that they get more funding to work on creating countermeasures

through these kind of things they're like it's impossible we're not going to stop

agentic swarms from developing and hacking so we need a lot more investment in the kind of

white hat agentic swarms of counter the hacking ones right and this should be a priority we need

to be investing in this we're ready to go if we can get more money so the companies are always

engaged in these like kind of self-serving things right or there were whistleblowers coming out

saying I worked at the company and I saw these practices and sometimes they're interesting insights

that those whistleblowers provide and other times you have to factor in that a whistleblower does not

automatically become super aware of everything and automatically correct there was a whistleblower

people have forgotten about them way back in the day I think he worked for what became Gemini

and he declared that it had consciousness and that it was like a conscious thing that he was

engaged with right and he was I think it was an ex-prest or like somebody who left the 7-year or

whatever and and that was a new story a year or two back we've now had access to the models

that he was interacting with and Gemini is regarded as like kind of stupid right out of the

existing crop of of models so you know I'm just saying sometimes people take that oh this is

an employee of anthropic or this is a senior executive at OpenAI therefore they know you know

the secret sauce and every comment that they make is like informed much better and the reality is

that like a lot of what they're talking about engineers know you know there there are open source

models that can perform very well right and and people are able to build them or run them on their own

PC so they're like it's not that there's no proprietary information but like in a lot of cases

it seems to be the amount of compute that people have or companies have the ability to do which is

their distinguishing feature but in any case I mentioned not just like consider the motives and

incentives that are that are going to run but so setting that all aside though might you know

the incentive some level of whatever if you'd like to continue listening to this conversation

you'll need to subscribe at patreon.com slash decoding the gurus once you do you'll get access to

full-length episodes of the decoding the gurus podcast including bonus shows grometer episodes

and decoding academia the decoding the gurus podcast is ad-free and relies entirely on listener

support and for as little as five dollars a month you can discover the real and secret academic

insights the ivory tower elites won't tell you this forbidden knowledge is more valuable than a top

tier university diploma minus the accreditation your donations bring us closer to saving western

civilization so subscribe now at patreon.com slash decoding the gurus