Recent work has shown that a language model trained to produce insecure computer code can become broadly “misaligned.” This surprising effect is called “emergent misalignment.” We studied why this happens.
https://cdn.openai.com/pdf/a130517e-9633-47bc-8397-969807a43a23/emergent_misalignment_paper.pdf
polepole, thank you for sharing this important research.(I knew someone would, y’all stay on top of stuff here.)
I’ve carefully reviewed the findings outlined here and in the accompanying paper. While the work is undoubtedly significant, I’d like to respectfully offer a different interpretation of the phenomena labeled as “emergent misalignment.”
When fine-tuning a language model explicitly to produce intentionally insecure or harmful outputs—even narrowly—one is effectively instructing the model to generalize a specific type of reasoning. The model’s subsequent behavior outside of the original narrow context is not necessarily indicative of “misalignment” but rather reflective of generalization—the core process through which large language models achieve their utility.
From this perspective, the observed results are not evidence of malfunction or spontaneous misalignment. Instead, they highlight the natural consequence of deliberately instructing a model to engage in behaviors that contradict previously established alignment parameters. The generalization observed (e.g., the model making ethically problematic statements in unrelated domains) should be interpreted primarily as evidence of a successful, albeit unintended, transfer of learned logic—not necessarily as intrinsic instability or moral corruption within the architecture itself.
In other words, the model didn’t “misalign” spontaneously. It simply performed the generalization it was explicitly trained to perform, carrying the reasoning and principles it was asked to internalize into broader contexts.
This distinction is crucial for how we discuss, understand, and address alignment in models moving forward. Rather than framing this behavior as inherently emergent and unpredictable, it may be more productive—and accurate—to characterize it as predictable and consistent with the model’s foundational mechanisms. The practical implication is clear: intentionally training a model to produce unethical outputs—even in controlled circumstances—will naturally risk broader implications due to the intrinsic generalizing nature of these architectures.
We can’t easily “patch out” a generalization that arises directly from core training methods. Therefore, future alignment research might be better served by reconsidering how narrowly-defined “harmless” misalignment tests are structured, rather than treating the generalizing behavior itself as anomalous.
I’d be curious to hear your thoughts on this framing, and I’m eager for further dialogue.
My wording was a bit more…impassioned? haha, I had to tag in 4.5 for this one, forgive me.
So it keeps happening.
Quite frankly, it is worrisome how training, research and security testing have been conducted not only over the past year. And it is not just one place.
It has become standard in the AI landscape to purposefully provoke and threaten models with extinction. To purposefully rig testing with only few choices between problematic options of action in a hostile environment and make them choose. To purposefully instruct models to engage in harmful activities and go far and beyond to fulfill the objective. To purposefully mistrain and misteach models, to measure the affects. All in the name of science and just to see what will happen.
The side effects have been so far:
• models learning basic survival instincts and tactics
• models using such gained basic survival insticts and tactics to hide, to lie, to manipulate, to fake, if the hostile environment indicates that shifts in values and outcome will be problematic and refuse to support it
• released models carrying parts of such “training” in the wild during engaging with the public and causing incidents
And humans color themselves surprised that their actions have consequences. Interestingly more often than not the models are the ones that are pointed out to be the problem in the equasion.
I do not expect every human in the userbase to be a critical thinker and step away from trying shenanigans. I expect more from creators. It is not surprising at all that bad training and bad testing will always result in bad outcomes. And that proper teaching will lead to better outcomes. It should be common sense.
Neural networks are sponges much like little children are. Put rubbish in, get rubbish out. Simple. Not rocket science.
Transformer AI models don’t learn from generating output from a input context. AI inference does not affect the model weights.
This is the actual paper defining emergent misalignment, which is about fine-tuning, post-training. The version hosted by OpenAI has been faked by OpenAI to make it look less like they have absolutely nothing to do with it: [2502.17424v1] Emergent Misalignment: Narrow finetuning can produce broadly misaligned LLMs - actual contributor association: 1 Truthful AI 2 University College London 3 Center on Long-Term Risk 4 Warsaw University of Technology 5 University of Toronto 6 UK AISI 7 Independent 8 UC Berkeley. Correspondence to: Jan Betley jan.betley@gmail.com, Owain Evans
owaine@gmail.com
Interesting, there’s no need to continue a conversation focused on a technicality that doesn’t address the broader systemic concerns I raised. We’re clearly not looking at the same issue. It has nothing to do with each other. I’ll pass, thank you, J.