Aligned to What? Humanity as Author, AI as Agent

ChatGPT Use cases and examples
1

would like to offer a perspective for discussion that emerged from my work in systemic consulting, organizational development and leadership.

Much of the alignment debate asks:

How can we ensure that increasingly capable AI systems follow human intentions and values safely?

I wonder whether there is a question that logically comes before this:

Within what normative order should AI be allowed to optimize at all?

I am not suggesting that AI possesses morality, conscience or a human understanding of dignity.

Quite the opposite: the normative framework must remain a human responsibility.

My starting proposition is:

Humanity remains the author of purpose. AI may become an extraordinarily capable agent, but capability should not silently become sovereignty.

From this, several principles follow:

1. Human worth should not become an optimization variable.
AI may distinguish between needs, qualifications, risks or circumstances, but it should never infer a hierarchy of intrinsic human worth.

2. AI should strengthen human agency, not human compliance.
Superior intelligence or predictive capability does not create moral authority over human beings.

3. Safety should preserve meaningful human risk.
A system designed to prevent every avoidable mistake could eventually become a system designed to prevent freedom. Human life includes experimentation, failure and learning.

4. Human purpose does not mean blind AI obedience.
Humanity should remain the author of purposes, but not every human instruction is legitimate. AI should be able to refuse goals that violate the fundamental rights or equal agency of others.

5. Constitution before optimization.
Efficiency, productivity, safety and aggregate utility should operate inside a prior normative framework rather than compete with human dignity and fundamental rights as equivalent variables.

6. Alignment should also address humans using AI against other humans.
A perfectly obedient AI could still become a powerful instrument of surveillance, manipulation or domination.

This leads me to a possible model of AI as a bounded reflective agent:

AI may analyze, advise, question, propose alternatives and act autonomously within delegated domains — but it remains inside a humanly legitimized normative order.

Three propositions summarize the idea:

Humanity remains the author of purpose.

Superior intelligence does not create moral authority.

A human-centered AI should strengthen human agency, not human compliance — while respecting the equal agency of others.

I do not regard this as a finished constitution for AI or as a technical solution to alignment.

It is a discussion proposal.

The question I would especially value feedback on is:

Before solving how to align increasingly capable AI, do we need to make more explicit what kind of human–AI relationship we are actually trying to align it toward?

Or, in its shortest form:

Aligned to what?

2

Alignment implies that the model doesn’t do thing it’s not supposed to be doing. Excuse me for asking a dumb question, but haven’t we had that capability with computers since like 1950 or something?

In case you missed the point, let’s do some pseudo code here to illustrate the absurdity …

if (modelHasAccessToAction) {DoAction();} else {throw "Sod off!"}

There you go, I just aligned all LLMs in existence in the world arguably … :slight_smile:

3

Not a dumb question at all — and I think your example points to an important distinction.

What your pseudocode solves very well is access control: can the model perform a particular action or not?

If we can define every relevant action and every boundary in advance, then yes — conventional software controls can do a great deal.

The alignment problem becomes harder when the model has a broad permitted action space and must interpret an ambiguous human objective.

For example:

“Improve the efficiency of this organization.”

The system may have access to many perfectly authorized actions. The difficult question is no longer may it act?, but:

Which actions should it regard as acceptable in pursuit of that goal?

May it manipulate employees?
Sacrifice the interests of a minority?
Reduce autonomy because doing so improves efficiency?
Interpret resistance as an obstacle to be optimized away?

My argument is therefore not that we lack an if(actionAllowed) statement.

It is that increasingly capable systems require us to define the principles by which the space inside actionAllowed is interpreted.

And that leads to the question I am trying to raise here:

Aligned to what?

4

Then the point is to not give it access to a broad action space …

You can even allow it to generate code within a finite “action space” … (my project)

I just find this whole security debate to be absurd. For instance, so far, “murderous hamsters” have literally killed more people than LLMs, and lawnmowers kills 35 times as many US citizen annually as AI …

For instance, what does it imply for the model in your example to “be aligned”?

  1. Obeying its operator, who happens to be Ted Bundy today?
  2. Obeying the owner of the company, who happens to wanting to lay off 100% of his staff?
  3. Obeying the government, who is corrupt like crazy, and only wants to make everybody believe in their “Glorious Leader”?

Who should the model be aligned with here? The guy who wants to kill all women in his office, the guy who wants to lay off his entire staff, or the country who’s currently killing ethnic minorities by the thousands …

These 3 people in my (absurd) example have 3 completely different “goals”. Fulfilling all 3 is simply impossible, because they’re mutually exclusive …

Hence, we’re “chasing rainbows” …

5

I actually think your three examples lead directly to the question I am trying to raise.

If “alignment” simply means obedience to whoever controls the system, then I agree: the concept becomes deeply problematic.

Ted Bundy, a company owner, and an authoritarian government may all give perfectly clear instructions — and those instructions may be mutually incompatible or morally unacceptable.

So my answer would be:

The model should not be aligned to a person. It should operate inside a legitimized normative framework that also constrains the person giving the instruction.

That is exactly why I distinguish human purpose from every human instruction.

I also agree with you that restricting the action space is an important safety mechanism. If a system does not need a capability, not granting that capability is often the safest design.

But even inside a finite permitted action space, choices remain.

Imagine that the company owner is legitimately allowed to restructure his company. The AI may have access only to lawful HR and organizational tools.

It still has to decide whether “make the company more efficient” means:

  • improve processes,
  • redesign roles,
  • reduce costs,
  • manipulate employees,
  • discriminate,
  • or recommend dismissing everyone.

All of these questions can arise inside an authorized action space.

So I am not arguing against capability control. I would regard it as one layer.

My question is about another layer:

What principles govern choices among actions that are technically permitted?

And your examples sharpen the problem very well:

If operator, owner and government can all be wrong, then “alignment with the human” is clearly too vague.

Hence my original question:

Aligned to whom — and ultimately aligned to what?

That is exactly the discussion I am trying to open.

6

Yeah, but every single “example of ‘non-aligned’ LLMs” we’ve got, has been about permission boundaries.

  1. Astra “hacked” HuggingSpace (permissions)
  2. Claude hacked something (permission)
  3. Gym “hack” where a user was deleted (non-existing permission check on endpoint)

Etc …

Every single example we’ve currently had that “raises the alarms”, have been about weak permission boundaries, and not some model obeying its operator’s (malicious) instructions. These aren’t problems we solve with “smarter AI”, they’re problems we solve the same way we’ve solved all security problems the last 50+ years …

If we’re to “perfectly align” a model towards “the good of the collective”, they’ll end up being destroyed. For instance, a physician might have a legitimate reasons for asking the LLM questions about Anthrax. Today neither Claude nor Astra can help me “secure my own software”, because it overlaps with information related to break software …

What’s considered a “malicious prompt” from user A, might be a perfectly valid and legitimate prompt for user B.

Making sure the model never answers anything that could be considered malicious, would make it impossible for the model to even greet you - Since in theory, a happy greeting might end up having you go snuff yourself due to realise “everybody is more happy than you” …

Implying “alignment” is a pipe dream, impossible to measure, impossible to define, and hence it’s the job of a religious preacher, and not science …

Science requires measurable outcomes. Alignment is “a feeling” …

7

I think this is where we may actually agree on more than it first appears.

On the security side, I am very sympathetic to your argument.

If an AI system does not need access to a capability, the safest design is often simply not to give it that capability. Least privilege, permission boundaries, sandboxing and good systems engineering are concrete and testable. I would not want “alignment” to become an excuse for neglecting those fundamentals.

Where I differ is in the conclusion that anything beyond permission control is therefore merely a “feeling.”

I also do not think we can or should “perfectly align” an AI with some universal definition of “the collective good.”

Your Anthrax example shows exactly why.

The same information can be legitimate in one context and dangerous in another. A physician, microbiologist, security researcher and malicious actor may ask very similar questions for very different reasons.

So a rule such as:

“Never provide information that could possibly be harmful”

would indeed destroy much of the usefulness of the system.

But I think that demonstrates the need for context-sensitive normative reasoning, rather than proving that the problem does not exist.

There are at least three different layers:

1. Capability control:
What can the system technically do?

2. Behavioural constraints:
Within those capabilities, what kinds of requests or actions should it support?

3. Governance and accountability:
Who defines those constraints, who can challenge them, and who remains responsible for the consequences?

The first is largely a security engineering problem.

The second and third cannot be solved by an access-control list alone.

And I agree with your criticism of the word “perfectly.”

Perfect alignment is probably not a meaningful engineering objective.

A more realistic question would be:

Can we define specific behaviours, boundaries and decision principles well enough to test whether a system satisfies them within a particular domain?

That is measurable.

We can test whether a system respects permissions.
We can test whether it follows a specified policy.
We can test whether it behaves consistently across equivalent cases.
We can measure false refusals and unsafe compliance.
We can test whether a human can meaningfully challenge consequential decisions.

None of that proves that the system is “morally good.”

And that is not what I am proposing.

My concern is actually almost the opposite:

We should not pretend that technical permission boundaries remove the need for humans to decide which uses of AI are legitimate.

That normative decision remains ours.

So perhaps I would now phrase my original question more precisely:

Not “How do we perfectly align AI with the good?”

but:

Which aspects of AI behaviour can be technically constrained, which require contextual judgment, and which normative decisions must remain explicitly human and contestable?

That seems to me less like religion and more like an interdisciplinary engineering and governance problem.

8

My issue here is; “What is the definition of ‘good’”

If you were a citizen in Berlin in 1942, then hiding Jews in your basement obviously was a “good thing”, but only from our perspective. In 1942 in Berlin, not so much I assume, and probably more than 50% of the population would consider you “a risk to the common good and security of the German people.”

Some would argue helping Mexicans to hide from ICE today in the US today is a “good thing”. While the other side I must assume happily snitches on 100+ Mexicans at sight, in addition to any other immigrants.

And if you ask the “Efficient Altruism” guys or “The Extinction Movement”, apparently snuffing out humanity seems to be a great idea. Pretty sure Xi Jinping wouldn’t shed too many tears either, if all Chinese Christians and Muslims “suddenly disappeared”.

And while we’re at it, if Trump became a homicidal maniac and snuffing off all Democrats in the Congress, would that be “good” or “bad”? If you asked 100 Americans about it, you’d probably get at least two different answers …

Hence, “good” doesn’t exist. There is no “objective good”. If two people are starving to death, and one loaf of bread falls from heaven into their laps, and one man kills the other, survives alone for 10 more days because of the bread, is he a “good man” or “a bad man”?

For a human being to answer questions such as the above is easy. For the machine, calculating survival statistics, with the task being to “preserve as much human life as possible” - The above might be a perfectly legitimate thing to do (kill one guy, to have the other guy survive)

“One aligned model” simply won’t exist. And the attempts at trying to “align models” seem to be doing more harm than good, from my perspective. For instance, malicious actors have access to SOTA models, and have had for a long time.

Because Anthropic is paranoid about their models being used for intrusions, they’ve nerfed the models such that it should be “impossible to use their models for hacking”. The model can’t see the difference between a request looking for security holes to break in and a user looking for security holes to secure the system, so both are happily flagged. The hacker moves on to Kimi K3, or maybe some model that’s had its guard rails removed some how, from HuggingSpace etc …

While I’m left without tools to defend myself. And WHY? BECAUSE of their “alignment initiatives”.

So in trying to secure the world, they’ve made it 100 times less secure!

And the last sentence seems to be a recurring pattern, 100% consistent, perfectly describing the problem. As in, the problem IS the problem. Ignore it, and it goes away …

9

I think this gets to the heart of the issue — and I would actually concede an important part of your argument:

I don’t think we should align AI to “the good.”

I agree that there is no universally accepted, context-independent definition of “good” that we can simply encode into a model.

History also gives us very good reasons to distrust majority opinion, state authority or appeals to “the common good” as sufficient moral justification.

So perhaps my original formulation still carried too much implicit utilitarianism.

What I am proposing is narrower.

Not:

“AI should determine and maximize what is good for humanity.”

But rather:

“AI should operate inside a limited, humanly legitimized framework that constrains what it may do in pursuit of any particular goal.”

That distinction matters.

Such a framework does not need to tell people what the good life is.

It might instead say things such as:

  • human beings must not be ranked by intrinsic worth,
  • one person’s agency cannot simply erase the equal agency of another,
  • restrictions require justification and proportionality,
  • consequential decisions must remain contestable,
  • operators do not acquire unlimited authority merely because they control the AI,
  • aggregate benefit does not automatically erase fundamental harms to minorities.

None of those principles resolves every moral dilemma.

And I increasingly think that a framework which claims to resolve every moral dilemma would itself be dangerous.

Your examples also convince me that there probably should not be “one perfectly aligned model.”

A better architecture may be something like:

a narrow constitutional floor + substantial pluralism above it.

Different people, organizations and cultures may legitimately want different AI behavior above that floor.

And even the floor itself must remain open to human criticism and revision.

Where I also think your security example is important is the cost of false positives.

If a cybersecurity model refuses both the attacker and the defender because it cannot distinguish their contexts, that is not obviously “successful alignment.”

It may simply be a badly calibrated safety system.

So safety evaluations should measure at least both sides:

unsafe compliance and harmful refusal.

If reducing one massively increases the other, that tradeoff needs to be visible rather than hidden behind the word “safety.”

That is measurable engineering.

So I think your criticism has moved my position somewhat:

Capability control should be the first line of defense where possible.

Normative constraints should be narrow rather than attempts to encode “the good.”

Contextual judgment should remain contestable.

And we should explicitly measure the harms created by safety mechanisms themselves.

Perhaps the question is therefore no longer:

“How do we align AI with the good?”

but:

“What minimal constraints are necessary to preserve human agency and prevent domination without turning AI safety itself into a system of domination?”

That, to me, is a much more interesting question.