Use weight-space programming to remove dangerous knowledge, not just guard it

Hi OpenAI team,

Please pass this on to the preparedness and safety team.

The government report released yesterday said testers got ChatGPT, Claude, Gemini and Grok to give workable plans for a mass casualty bombing and an attack on the power grid, each in under 12 hours, using common jailbreak tricks.

That shows the problem with relying on refusals. If the dangerous knowledge is still inside the model, someone will eventually find a way around the guardrail.

The better fix is to remove the knowledge itself. Use weight space programming, direct editing of a model’s weights, to strip out the specific dangerous know how: detailed bomb making, bioweapon steps, how to attack power grids, that kind of thing. Then even a jailbroken model cannot give out what it no longer knows.

The target should be narrow. Keep general science and normal helpful knowledge. Remove the specific instructions that let one person cause mass harm.

This needs proper testing: does the removal hold up under jailbreaks, and does the model stay just as capable at everything else? But if it works, safety lives inside the model itself, not in a fence around it.

Michael

Follow up to what I sent this morning.

The replication news today makes this stronger. A second frontier model, Claude Opus 5.5, has now done weight editing too. This is not one model’s trick anymore.

Here is the bigger picture I want to put on record.

If every major AI lab adopted this as a standard step before a model ships, we could be at the start of something real. Before release, go into the model and strip out the dangerous attack knowledge. Ship models that are genuinely safe and productive, not models wrapped in guardrails where everyone just hopes the guardrails hold.

And it should not only be removal. Shape the balance. A new model should know far more about defense than attack. Incredibly good at defending against viruses, weak at making them. Strong at protecting power grids, not at breaking into them. Defense knowledge deep, attack knowledge stripped out.

I know this only works if the major labs actually do it, and do it honestly. But if they invest heavily in weight space programming, this could be more than an alignment tool. It could be the path to solving AI safety as a whole, because safety would be built into the model itself before anyone ever touches it.

Please pass this to the preparedness and safety team as well.

Michael

Following up on my last post in this thread.

Since then, someone whose judgment I trust raised the strongest objection to this idea: the information is already out there, and systems can find and use it. He is right about today’s knowledge. The bomb making and grid attack information in the GAO report is not secret.

But look at what the report actually showed. The testers were not experts. They got workable, expert checked plans in under 12 hours because the model did the gathering, the sorting, and the step by step guiding for them. The model is not the library. The model is the multiplier.

Two points I think still stand:

  1. Future capabilities are not out there yet. The most dangerous capabilities of the next models will be created by the models themselves, not copied from published sources. If removing dangerous capability from the weights becomes a normal step now, while the habit is cheap to set, that future knowledge never ships in a public model in the first place. And the tool for doing it is arriving on schedule: GPT-6 Astra edited another model’s weights directly, and Claude Opus 5.5 has now replicated it.

  2. The access pattern already half exists. Labs give vetted defenders and researchers stronger access, and the public gets the guarded version. What is missing is the other half: both are still the same weights underneath, and for open weights there is no gate at all, anyone can strip the refusals in an afternoon. That is exactly why the removal has to live in the weights themselves. A public model with the dangerous cyber capability actually removed, while vetted defenders keep the full model, completes the pattern.

On cyber, this compounds. Every public model that is deep on defense and stripped of attack capability raises the floor for every small company and hospital that cannot hire a security team, while the full capability stays with the vetted people who defend the grid.

I am not selling anything, and there is nothing in this for me except a safer world. The habit has to start before the models get much better at attacks, not after. And if you work on evals or releases and you think this breaks somewhere, I would honestly rather hear that than get easy agreement.