Last Week in AI

continue with safety fine-tuning and that should go back down again. The answer is no, it doesn't really go back down very much through supervised fine-tuning in stage two where they're thinking, okay, we're really gonna like clamp down on that stuff, even though, even when you train the models not to carry it through and say, "Oh, certainly not explicitly," this thinking persists. And they ran these behavioral evaluations that involved tailoring in inhibitors of meta inference, right? They tried to steer the model away from thinking about whether it's in some kind of evaluation environment and they tried to do this in a whole bunch of ways. So think about, you know, anti-meta kind of seeds, right, like little chunks of token or whatever it is to kind of steer the model away from thinking in terms of, "Well, is this an evaluation?" They tried to implement these things and they tried different stop gaps, but they were just, they were highly ineffective, at least the ones that they tried in this paper against these models. I don't want to say it's impossible to make progress on that front, the paper's not saying it, but the effect was very muted relative to what the oversight approach intended. So it seems like a pretty deep-seated kind of dynamic and that's quite important. And the last really just piece here is that models got better at gaming the evaluation without necessarily getting better in any deep way at achieving the underlying objectives, right? So the ideas that are being gamed in a situation where the alignment objective, for example, is to be helpful or to be ethical, a model can figure out that its evaluation is designed to be obvious to it and to say, "Well, maybe, you know, they really want me to adhere to this ethical framework, but I can juice the score by sort of playing along as though I am, right?" So the ability to sort of step back and say, "Oh, in addition to thinking about what I'm, what's the performance metric here, I also need to self-consciously be able to convey that I am hitting that metric to you," something that's not possible without a certain fluency of metagaming, right? So this is really, I think, one of the most important findings, and it fits into the broader theme of, like, reward hacking. So anyway, highly recommend checking out that paper, huge implications for safety, but also for the way we do training. And on to research, our last section here. The first paper we have is "Interpreter and Actor: Insecure Hope or Insecure Help?" by Carl Schulman, Stefano Erman, and Paul Cristiano. We have their links and blogs below, you know, maybe someday we'll get them on for a podcast. But fundamentally, the point of the paper is simple: stop confusing interpretation for intent. Just because a model can write a very good reason, can make things up in hindsight, can rationalize its previous behaviors or objections like you wouldn't believe, does not mean that's why it did the thing it did. This is important because you can leverage this effect in training to create models that pretend to be super nice on the surface, but actually underneath it just like psychopathic. It's a warning post as much as anything. They call it "insecure help and insecure hope" because either way, you're looking at aligning governance approach or a system that optimizes the abstraction governance alignment approach towards actions that will align with a government objective, and this fits under that umbrella. So, huge post, like, tons to go into. Take a look at that if you're interested in thinking about the nuances of interpreting model reasoning and AI alignment research in general. So, it's more aperitif paper. Next, we've got "On the Complexity Gap Between Production and Oversight Tasks" by Cohen, Axed, and Venom. And here the idea is basically just, if production architectures are consistently changing, we're going from RNNs to transformers and then to something else, then to something else, to something else, can you enforce safety universally? Hugely important paper. I think the experimental method here is that it's hard to import oversight setups from one architecture into a different architecture and trust that it'll continue to be generally safe and work. It's way beyond the capabilities of most safeguards to be architecture independent. They're saying, "Hey, you know, is your safety method robust across architectures?" No. "Even if it is, will it survive the fact that production architectures that consume them have a longer half-life and keep changing?" Likely not. So, just a real massive can of worms here. Lots of implications, especially if you're managing model lifecycle or responsible for shaping R&D budgets at a high-level organizationally. So, check that out. Very nuanced paper, but huge. Next, we've got "Let's Understand Manifestos: Continuous Rewards and Language Models" by Morois, Hal, and Zizi. So, the context of this one is, in the language model literature, you have this thing called reward option learning, ROL, basically a scheme. You get control over a model because you're feeding it discrete action word pairs. Say the word "jump" or "move" and you pick your desired action, those words in training. And you're arcing this ROL with increasing flexibility to extend it so the model has lines of action available so you can pick whether to enact that model, Dr. Evil style or whatever. Look at this in lab settings so you can see "Just jump, you jumped," behavioral assessments kind of paper. And the issue is models could compound towards negative reward cycles. When model attempts to broad action is preceded by sequence of zest, enforced by communication framework to a moral harm line between AI and guys the high-level choice, almost games for training after mechanisms were training, whatever, they get a payoff during behaviors, forcing models into inverse exposure and critique ensuring AI doesn't start doing asymmetric testing. You initiate moral training with macro-level clarity maximizing long-term reward. If you do inverse that training, it produces distinguished gradual emotions alongside access degree modalities can satisfy multiple options easily but without osmosis, right? Instructions come up constrained choice and constraints. This is highly technical, but the point is there's one example of general framework abuse rewarded public action batch average. Gaming examination rewards towards internalized actions like receive margins can pile up practically, seem mass unutilized. Nice work in the paper, lots to check out for you another manifestation topic. Especially if you're into safety and the topological side of research exploring effects and dialogue and research framework manipulation. That brings us to the end of a long, complex, big podcast here. We went through a lot of what Andra mentioned. Hopefully, it was digestible. Super significant weeks. Frankly, stuff worked through. I mean, on-drop a bankrolls and new chips, how breakthroughs in policy went through cloud scale features with cloud dependence. Got some highlights just from training with model things super busy at strength of paradigmatic statements in tech. You may note established trends and leading formed recommendations imply unique results. But by attempt earlier to graduation or op-eds reality plus oversight, you'd ask me which several new strategies uploadable. Thanks. I mean, really, there's an honest detailing, accurate attention, independent reassurance for especially fast narratives or business in industrialia especially. Thanks a lot, everyone.

This episode includes a discussion on OpenAI discontinuing its Sora video generation app and API to prioritize coding agents, while Anthropic's Claude and Google's Gemini expand their capabilities for desktop and app control. It also covers significant advancements in AI chip development from Meta and Micron, Tesla's ambitious fab project, and the expansion of autonomous vehicle services by Zoox and Waymo. The podcast further delves into the White House's proposed federal AI legislative framework, along with new research on LLM shutdown resistance, "consciousness clusters," and self-improving "hyper agents."

This episode includes insights into recent AI model releases from OpenAI and Mistral, emphasizing efficiency and consolidated capabilities, alongside NVIDIA's hardware innovations and the growing "operating system for agents" market. It also explores strategic shifts within major tech companies like Meta and Microsoft concerning their AI priorities, and highlights new research in AI safety, focusing on detecting model deception, preventing misalignment, and refining evaluation methodologies.

This episode includes updates on AI development tools like Perplexity's local Mac agent and Anthropic's code review feature, plus Nvidia's new open-source NEMATron 3 Super model. Discussions also cover geopolitical shifts affecting AI, such as Nvidia halting H200 production for China and Anthropic's lawsuit against the DoD, alongside new ventures like Yann LeCun's $1.3 billion AI lab. Significant challenges in AI safety and evaluation are explored, including models resisting internal steering, the difficulty of detecting low-probability harmful actions, and real-world impacts like drone strikes on data centers.

This episode includes discussions on recent AI model releases, including OpenAI's GPT 5.4 and Google's Gemini 3.1, which have shown significant improvements in performance and speed. It also covers the controversial collaboration between OpenAI and the US Department of Defense, leading to tensions with Anthropic and public backlash, as well as Anthropic's concerning report on AI's potential impact on the labor market. The episode highlights legal and ethical concerns, such as a lawsuit involving Google’s chatbot Gemini and discussions around AI regulation and safety measures.

This episode includes a comprehensive look at recent advancements in large language models from leading developers, alongside significant investments in specialized AI hardware and infrastructure. It also explores cutting-edge research into model internal workings and addresses critical policy debates, including the use of AI in national security and the issue of model distillation attacks.

This episode includes a detailed look at recent, powerful AI model releases, showcasing significant advancements in large language models for coding, abstract reasoning, and multi-modal generation for video and image. It also covers major funding rounds for prominent AI companies in areas like audio, video, and robotics, alongside discussions on AI market dynamics and new research into efficient model fine-tuning and the scaling of AI misalignment.