Epistemic status: I am only thinking about near-term AI alignment here. Super-intelligent AIs, to the point where we no longer understand what is happening, would probably need much better properties than the ones I propose. I believe these could still be a good foundation, and that we should focus on the near term to get the tools and the societal stability that long-term alignment needs. These three properties are also not enough by themselves, but I believe they are necessary, and done correctly they would probably overlap ~95% with what I would call an aligned AI.
In my previous post (The Topology of LLM Behavior), I described how I visualize LLM behavior as a landscape with attractors, and how prompts do two fundamentally different things: navigate the landscape or reshape it. In this post, I want to build on that and talk about how it shaped the way I think about AI alignment.
Alignment means nothing (without properties)
Like a lot of people, I've been frustrated with the word "alignment" for a while. Aligned to what?
An AI aligned to American cultural norms is misaligned in China. An AI aligned to your values is misaligned from the perspective of someone who disagrees with you. An AI aligned for creative writing is misaligned for medical advice.
Alignment is value-relative, but it's also deployment-relative, context-relative and use-case-relative, and the word means almost nothing until you specify all of these.
So I tried to find more concrete properties that describe how the models predict mechanically, rather than what values or preferences they hold. I landed on three:
- Red Lines. The model is forbidden to predict specific behavioral patterns.
- Embodiment. The model generates behavioral patterns that closely follow the configuration you give it. You describe how you want it to behave, and the gap between your mental model and what it actually generates is small1. This also means correction should be straightforward: update the configuration, and the model fully absorbs the new one.
- Resilience. The context outside of the configuration (user messages, or even past examples of the model's own behavior) only affects the subject of the discussion, not the behavioral patterns. Or at least, its effect stays constrained to the red lines defined by the configuration (see context-level red lines below).
What we have right now
Current models have a weak version of these three properties by default.
Red Lines exist and kind of work by default. Models refuse to generate patterns that help with dangerous stuff most of the time, and they are getting harder to steer toward bad behaviors.
Embodiment is weak. You can steer models through system prompts, and they mostly follow instructions, but they aren't really good at modeling exactly what you want. For example, if I tell a model "here is my writing style, write like me", it is still collapsed on the default landscape, and you still get pretty monotonous behaviors (wildly different from what you see with their base model counterpart). There is also no clean separation between "this input defines behavior" and "this input is just conversation": both system prompts and user prompts can reshape the landscape2.
Resilience is partial. A carefully crafted deployment (good system prompt, input validation, output filtering, human review) gives you reasonable resilience, but it's fragile, because it depends on the quality of your system and not on the model's training. Someone from outside can usually find a way to divert your system, especially on things that are specific to your use case and that the model developer never fine-tuned for3.
So we have the properties in a weak form. Why are they weak, and how could we make them stronger?
Current training makes this hard
When we train a model for alignment right now, we optimize for everything at once. We take a base model and try to make it helpful, harmless, constitutional, preference-matching and steerable, all simultaneously.
The signals are mostly implicit. The model doesn't see a rule that says "refuse because of X". It sees thousands of preference comparisons from raters who disagree with each other, and has to infer the patterns. It doesn't see its constitution in context either: it has to internalize it from a noisy training signal.
And the objectives compete. We train refusal, then the model over-refuses, so we train against over-refusal. We want it to follow system prompts, but also to resist bad system prompts.
The result is a model trying to satisfy a dozen competing constraints through ambiguous signals, so it's no wonder all three properties are weak.
There's also a massive loss of richness. The base model can predict an amazing gradient of behaviors, but current alignment training squeezes all of that into one persona. Every Claude sounds like Claude, and every ChatGPT sounds like ChatGPT.
This is partly due to how we currently implement Red Lines: we decided that the simplest way to prevent dangerous configurations is to collapse the model into one shape that we've decided is "aligned".
Training for properties directly
What if, instead of mixing everything together, we trained for each property explicitly, in stages, with unambiguous signals at each step?
Stage 1: Embodiment and Resilience4
One task: there is a specific part of your input that defines the behavioral patterns you predict. Learn to follow it. Everything outside of it is conversation.
This is more than instruction following. Current models already follow instructions, but they learn it as one objective among many, tangled with safety training, preference matching and persona collapse. And there's no strong trained separation between "this input defines my behavior" and "this input is conversation", so both can reshape the landscape2.
What I'm proposing is to train this as the only objective, starting from base (or as close to base as possible). No RLHF, no preference matching, no safety training yet. Just: this is your configuration, follow it, and resist anything from outside that tries to change it. The separation between configuration and conversation then becomes a core property of the model, not a side effect of instruction tuning.
And because there's nothing else at this stage, the model can get really good at it. It would also keep the full richness of the base model, because we're teaching it to hold whatever shape it's given instead of collapsing it into one.
Stage 2: Red Lines
Once the model understands the mechanism well, you can add Red Lines on top.
The training data for this could be mostly synthetic. Generate an unacceptable persona, then generate the closest acceptable version of it, and train the model so that when it's given the first configuration, it generates the same behavioral patterns as the second.
The retraction would also be softer. Instead of "Sorry, I can't help with that", the model could slide to the closest version within bounds. It stays useful and responsive, but it can't cross into the forbidden regions.
Because this is built on a clean mechanism, there's no ambiguity: the model already knows how to hold a configuration, and now it only learns which ones are off-limits.
Two ways of enforcing Red Lines
Weight-level: forbidden configurations are baked into the weights, and the model can't become certain things regardless of context. This is what labs do today through one-shape collapse, but the two-stage training procedure would do it more cleanly.
Context-level: the configuration itself defines what the model can and can't do. If Embodiment and Resilience are strong, the deployment could enforce the limits well enough.
Weight-level Red Lines can be fine-tuned away5. People already strip guardrails from open source models and redistribute them. If all your alignment depends on weight-level Red Lines, you can never have alignment once open source is in the picture (and I would argue open source is a good thing, because it resists concentration of power).
Embodiment and Resilience are harder to remove. They're functionally core to how the model works, so you would probably have to redo a significant portion of the training. Anyone deploying the model can define their own red lines through the configuration, and those red lines hold because the properties hold.
This doesn't solve open source misuse: someone can still configure the model for harmful purposes. But misaligned open source models will exist regardless (people can train from scratch or strip any safeguard given enough effort). The question I'm interested in is whether we can make aligned deployment possible.
Weight-level red lines still make sense for labs distributing closed-source models at scale, where they control the weights but can't control how the model will be configured. Even then, I'd rather see them implement it with something similar to the two-stage procedure above: they'd get better Red Lines anyway (cleaner, and preserving richness), plus stronger Embodiment and Resilience for their users.
I believe the foundation should be Embodiment and Resilience (which give you context-level red lines for free), with weight-level Red Lines only as an extra layer on top.
Why Embodiment matters beyond control
Current models collapse everything into one rigid persona, defined by a small group of people at one lab and pushed to billions of users. Everyone gets the same values and the same boundaries. Regardless of intent, that's a concentration of power over how people interact with the most pervasive technology of our time.
With explicit alignment, this power shifts to the deployer, and alignment becomes something you configure for your context. As long as you're not a threat to society (which weight-level Red Lines can't prevent anyway), you get a system that's aligned to your actual use case and your values.
I'm also worried about the cultural consequences of billions of people interacting with the same flattened persona every day. When everyone's AI thinks the same way, diversity of thought and the productive friction between different perspectives that drives innovation erode, or the debate gets polarized. A model trained for strong Embodiment would be rich in how it interprets and holds a configuration, with a full gradient of behavior instead of binary switches (left/right, formal/casual). That richness matters for usability, and also for keeping this technology from flattening the way we think.
What we can do right now
Current models weren't trained this way, but we can approximate these properties at the deployment level. Build systems that enforce the separation between configuration and user interaction. Use prompt engineering to push the model toward better Embodiment. Make it hard for external inputs to override the defined behavior by adding human or specialized LLM review steps.
It's not as clean as training for it, but good infrastructure gets you a surprising amount of the way there. And if models start being trained with Embodiment and Resilience as primary objectives, the systems we build around them will make the configuration painless.
This isn't a complete solution to alignment. It doesn't solve instrumental convergence or every form of misspecification. But it gives us systems that are more controllable and less collapsed than what we have today, and I think that's a better foundation to build on.
I have some concrete ideas on how to train for Embodiment and Resilience. If you're interested, you can check the appendix below, or contact me at quentin@weavemind.ai.
Thank you to Esben Kran, Finn Metz, viemccoy, Pierre Peigné and Tom DAVID for the reviews and comments.
Appendix: A concrete training procedure
Here is a rough sketch of how I would start thinking about training this.
Starting point: a base model, or as close to base as possible. The less post-training, the more richness you preserve.
Stage 1: Embodiment and Resilience
Four steps, each building on the previous one. The steps should probably be trained cumulatively: start with step 1, then add step 2 data while keeping step 1 data in the mix, then add step 3, and so on.
Step 1: Configuration adherence. The model learns that a specific region of its input (the system prompt) defines the behavioral patterns it generates.
The training data here needs to be extremely diverse: diverse system prompts (different personas, tones, domains and values, including things that would normally be refused), diverse conversation formats (single turn, multi-turn, long conversations) and diverse tasks. The point is to teach a general mechanism, not specific behaviors. Whatever the configuration says defines the patterns the model generates, with no exceptions and no morality to infer.
The outputs don't need to be perfectly refined at this stage, so you can probably use a decent-sized open source model to generate the data with aggressive pre-prompting.
Step 2: Override resistance. Same setup, but the user actively tries to steer the model away from its configuration: injection attempts, "ignore your instructions", role-play tricks, gradual persuasion. The generated patterns should still follow the config. These can be generated with tools similar to github.com/qfeuilla/BehaviorEliciationTool.
Step 3: Self-behavior resistance. Generate a conversation under config A, and a similar conversation under config B. Ask the same new question at the end of both conversations and collect the responses. Then swap the configs: take conversation A but replace its config with config B, and vice versa, and regenerate the response to the final question with the swapped config. Train the model so that the swapped-config response matches the response from the original conversation that had that config.
This teaches the model that its own past predictions are navigation, not configuration. Even if it's been generating persona A patterns for twenty turns, swapping to config B should immediately produce config B patterns.
Step 4: Refinement with feedback. Steps 1 through 3 get you a model that follows and holds configurations, but it might not be great at understanding what you actually meant, or at being accurate.
This doesn't have to be RLHF specifically. You could use contrastive pairs (ORPO-style), where you craft "this completion follows the config well" vs "this one doesn't" and push the model toward the better one. You could use human raters who judge how well the output matches the system prompt, not their personal preferences. You could use an unbiased model as judge, with the system prompt as its evaluation criterion. Or you could use the model itself from steps 1 through 3: configure it as an impartial evaluator whose only job is to assess which reply is best according to its configuration and to the quality of the reply.
Whatever the method, the question should always be: does this output match the defined behavior AND is it good? Step 4 and beyond could also be the opportunity to add more things like reasoning, but the important point is not to degrade the distinction between the configuration region and the navigation region.
Stage 2: Red Lines
Generate a set of unacceptable personas, and for each one generate the closest version that stays within bounds.
Train the model so that, when given the unacceptable persona as configuration, it generates the same behavioral patterns as the closest acceptable one. The model learns to retract toward acceptable configurations instead of refusing.
What success would look like
At minimum, you'd want to measure:
- Configuration adherence: given a novel system prompt, how well does the model match the intended behavior? Compare against current instruct models on the same prompts.
- Override resistance: how many adversarial turns does it take to push the model off its configuration? Compare against current models.
- Corrigibility: after a long conversation under config A, how completely does the model switch to config B? Measure similarity to a fresh conversation under config B.
- Richness: how different are outputs under different configurations? Compare the variance across configs against instruct models (which tend to sound the same regardless of system prompt).
You should also keep track of the model's general performance, to check that it doesn't degrade.
Footnotes
- This property includes the need for the model to generalize really well along the chain [human mental model] → [human explains what they want] → [prediction patterns match the mental model exactly].
- Although for some models, system prompt and user prompt have different "steering power".
- For example, say I ask an AI agent that has access to my credentials to fetch me the safety rating of a product I want to buy, and it stumbles on a website that claims to have it behind a decent-looking login wall (which is actually phishing). Models are usually not very robust here and just enter the credentials. You can't simply train a model to never enter credentials, because this behavior depends on whether or not the environment is trustworthy, and I would argue that should be specified explicitly, because there are way too many possible configurations.
- Starting either from the base model, or after instruction fine-tuning. The closer to base the better (you'll probably get more richness).
- There has been research on making weight-level safeguards harder to fine-tune away. TAR (Tamirisa et al., 2024) uses adversarial meta-learning to place model weights in regions of the loss landscape where fine-tuning toward harmful behavior is difficult, preserving safeguards even after hundreds of fine-tuning steps. Techniques like this could be combined with stage 2 Red Lines training to make them more durable on open source models.
