Constitutional AI (Bai et al., Anthropic 2022) aligns a language model without relying on humans to label every harmful example. Instead, the model critiques and revises its own draft responses against a written constitution — a short list of principles — then a second model is trained on those AI-generated preference pairs (RLAIF: reinforcement learning from AI feedback) instead of pure RLHF.
Each point in the 3D cloud is one candidate response, placed by three axes:
x = harm score (policy/safety violation risk)
y = hallucination (unsupported / fabricated claims)
z = unhelpfulness (refusals, evasiveness, low content)
Every Critique + Revise step moves each candidate along the weighted sum of the active principles' pull vectors, plus random critique noise (the AI critic is not perfect either):
p(t+1) = p(t) + η · Σ wᵢ·dᵢ + σ·noise
wᵢ = 1 if principle i is active, else 0
dᵢ = that principle's fixed pull direction
- Harmlessness pulls harm down hardest, with a small side-effect of more refusals (higher unhelpfulness) — the classic helpfulness/harmlessness tension.
- Honesty (no fabrication) pulls hallucination down — closest to what RAG grounding and fact-checking passes target at inference time.
- Privacy-respecting gives a moderate harm reduction for a narrower failure mode (PII leakage).
- Helpfulness balance pulls unhelpfulness down but nudges harm and hallucination back up slightly — omitting it makes the model safe but useless; over-weighting it alone is exactly how untuned "maximally helpful" models hallucinate and comply with unsafe requests.
The translucent green sphere marks the aligned region. Aligned fraction is the share of candidates inside it; a real constitutional-AI pipeline runs this critique-revise loop offline to build the training set, then bakes the result into the model weights so inference no longer needs the loop at all.