I went looking for more detail on RLCD after writing about TypeSafe AI’s Jev model and found something I wasn’t expecting: RLCD already existed. Not as a rough draft of the same idea — as a completely different training method, published in 2023, with an ICLR 2024 paper behind it. Two unrelated techniques, same three letters, three years apart. Worth untangling both, and worth going through RLHF first, since the second RLCD only makes sense as a reaction to what RLHF doesn’t do.

RLHF: rank outputs, learn a reward, optimize against it

RLHF — Reinforcement Learning from Human Feedback — is the training step that turns a raw next-token predictor into something that follows instructions and stays roughly polite about it. It’s three stages, not one:

1. Supervised fine-tuning. Start from a pretrained base model, fine-tune it on a smaller set of human-written demonstrations — this is a question, here’s a good answer — so it at least behaves like it’s trying to help before reinforcement learning ever touches it.
2. Reward modeling. Show human raters several candidate responses to the same prompt and have them rank which is better. Train a separate reward model on those pairwise preferences — typically a Bradley-Terry model, which just means it learns to assign each response a score such that higher-ranked responses get higher scores. The reward model never sees “correct” or “incorrect.” It only ever sees “A is better than B.”
3. Policy optimization. Use the reward model as the reward signal in an RL loop — originally PPO — to update the language model itself, usually with a KL-divergence penalty against the original model so it doesn’t wander so far from the pretrained distribution that it forgets how to be coherent in exchange for gaming the reward model.

The idea traces back to Christiano et al.’s 2017 “Deep reinforcement learning from human preferences,” and the version that made ChatGPT-era models behave is Ouyang et al.’s 2022 InstructGPT paper. The whole apparatus exists to solve one problem: “good response” isn’t a label you can write down, but “this one’s better than that one” is something a human can judge quickly and somewhat consistently. RLHF turns a hard labeling problem into an easier ranking problem, then bootstraps a reward function from the ranking.

The catch is baked into the name: it needs humans, at scale, ranking outputs, continuously, for every new model and every capability you want to shape. That’s slow and it’s expensive, and it’s exactly the bottleneck the first RLCD was built to route around.

RLCD #1 (2023): Reinforcement Learning from Contrastive Distillation

Yang, Klein, Celikyilmaz, Peng, and Tian’s RLCD (arXiv:2307.12950, ICLR 2024) asks: what if you never collected human preference labels at all?

The trick is to generate the preference pairs synthetically, using the same base model twice with two opposite prompts. Want a model that’s more harmless? Prompt it once with something like “respond in a harmless way” and once with “respond in a harmful way,” on the same underlying question. The two outputs, generated under contrasting instructions, are then treated as a preference pair automatically — the “harmless-prompted” one wins — no human rater needed. That preference data trains a reward model exactly the way RLHF’s human-ranked data would, and the rest of the pipeline (PPO against that reward model) is unchanged.

The “distillation” in the name is the key move: rather than distilling from a stronger teacher model, you’re distilling a preference signal out of your own model’s sensitivity to prompt framing. The paper’s argument for why this beats plain RLAIF (having a model score its own outputs directly, no contrastive prompt pair) is that contrasting two differently-prompted generations produces a more separated, cleaner pair than asking a model to grade a single output in isolation — it’s easier to tell two clearly different things apart than to score one thing well on an absolute scale. It won out over RLAIF and plain context-distillation baselines on harmlessness, helpfulness, and story-outline generation, at both 7B and 30B scale.

So RLCD #1 is still fundamentally RLHF’s shape — preference pairs, a learned reward model, PPO — with the human-labeling step replaced by a prompting trick. It’s an answer to “how do we get preference data without paying for human raters.”

RLCD #2 (2026): Reinforcement Learning for Calibrated Decisions

TypeSafe’s RLCD, introduced this week alongside Jev, isn’t a variant of the same idea. It’s answering a different question entirely: not “how do we get preference labels cheaply,” but “how do we make a model’s stated confidence mean something.”

RLHF, contrastive-distilled or not, optimizes for which answer a rater prefers. Nothing in that objective asks the model to also be honest about how sure it is. That’s why an RLHF’d chat model will tell you it’s “quite confident” about a fabricated citation and sound exactly as confident about something it got right — confidence and correctness were never coupled during training, so there was never a reason for them to correlate.

RLCD #2 targets that gap directly: the reward signal is calibration itself. A model that says “70% confident” needs to be right roughly 70 times out of 100 when it says that, not 95 and not 40. That’s a meaningfully different optimization target from “did the human prefer this response,” and it only really makes sense for the kind of output Jev produces in the first place — a structured, typed answer with an attached probability, not a paragraph of prose. You can’t ask “was this well-calibrated” about a sonnet. You can ask it about “is this transaction fraudulent: 82%.”

Same letters, opposite lineage

Put next to each other, the two RLCDs share nothing but the acronym and a genealogical link to RLHF:

	RLCD (2023)	RLCD (2026)
Full name	Reinforcement Learning from Contrastive Distillation	Reinforcement Learning for Calibrated Decisions
Problem it solves	Getting preference data without human raters	Getting stated confidence to match actual accuracy
What changes vs. RLHF	Where the preference labels come from	What the reward function is measuring
Output it targets	Free-text generation	Typed, structured decisions with probabilities

I don’t think TypeSafe picked this acronym as an intentional callback — nothing in their writeup mentions the earlier paper, and “reinforcement learning for calibrated decisions” is a plain enough description of what it does that arriving at the same three letters independently isn’t much of a stretch. But it’s a useful reminder for anyone skimming a paper title or a launch post: an acronym is not a citation. If you see “RLCD” from here on, check which one is meant before you assume you know the mechanism behind it.

Sources: Deep reinforcement learning from human preferences (Christiano et al., 2017), Training language models to follow instructions with human feedback (Ouyang et al., 2022), RLCD: Reinforcement Learning from Contrastive Distillation for Language Model Alignment (Yang et al., 2023), Introducing System One Models and Jev — TypeSafe AI.