JD
You
Walk me through how Constitutional AI differs from standard RLHF, in two paragraphs, for a reader who has trained models but not read the paper.
C
Claude
Standard RLHF asks human labelers to rank model outputs, then trains a reward model on those rankings, then fine-tunes the policy against that reward. The signal is dense but opaque, the model learns "this kind of output gets high reward," not "this kind of output is honest."
Constitutional AI replaces the human ranker with the model itself, evaluating its own draft against a written list of principles. The model produces a critique, then revises. The signal is sparser but the rule it learned is now legible, you can read the constitution and predict the refusal.
Constitutional AI replaces the human ranker with the model itself, evaluating its own draft against a written list of principles. The model produces a critique, then revises. The signal is sparser but the rule it learned is now legible, you can read the constitution and predict the refusal.