Process vs outcome rewards
ai
rl
reasoning
reward-modeling
grpo
Before you tune a single hyperparameter on a reasoning RL run, you've already made the decision that matters: do you reward the answer, or do you reward the steps. One of those choices barely trains on long chains. The other one is the easiest thing in your whole stack to cheat.
Everyone wants to collapse this into "PRM good, ORM bad" or the reverse. Neither is right, and the discourse skipping the actual tradeoff is why people keep shipping the wrong one. Let's derive it.
A reasoning trajectory looks like s_1, s_2, ..., s_T -> y. Steps, then a final answer. Somewhere in your training loop you compute a reward and push a gradient through the policy. The fork is: does that reward live on y alone, or on every s_t along the way.
That's it. That's the whole decision. Everything downstream (how sparse your signal is, how fast the model learns, how badly it can be gamed) falls out of this one choice. People treat it like a knob you tune after you've picked your architecture and your data mix. It's not a knob. It's upstream of the loss function.
Outcome reward models (ORM) score y. Process reward models (PRM) score s_t for every t. Both are trying to answer "was this good," they just disagree on the unit of "this."
The setup is brutally simple. A verifier checks the final answer and returns a reward R(y) that is either 0 or 1. Every step gets r_t = 0 except the last one, which gets r_T = R(y). With a baseline b, the policy gradient signal on every single token in the trajectory is the same number:
advantage on every token = R(y) - b
Read that again. Whether the model is scoring the gradient on step 1 or step 47, it's multiplying log pi by the exact same scalar. If the answer's right, every step gets pushed up. If it's wrong, every step gets pushed down. There's no mechanism anywhere in this math for "step 12 was fine, step 34 was garbage." The model doesn't know which steps carried the trajectory and which ones almost sank it. It only knows the boat arrived.
This is honest. There's no way to fake R(y) if your verifier is real (unit test passed, math answer matches, proof checker accepted). It's also brutal credit assignment: one Bernoulli sample has to explain a 40-step chain of reasoning. High variance, by construction, because the thing you're trying to learn from is a single coin flip stretched across a long sequence.
A process reward model gives you a value V(s_t) at every step, not just at the end. The natural thing to build from that is a TD-style residual, localized to step t:
delta_t = r_t + gamma * V(s_t+1) - V(s_t)
Now the gradient on step t depends on whether that step moved the estimated value in the right direction, not on some scalar computed 30 steps later. If step 12 improved your position and step 34 made things worse, delta_12 and delta_34 can have opposite signs even inside the same successful trajectory. Credit is assigned, not smeared across the whole chain.
This is the entire appeal. It's also, as we'll get to, the entire problem.
Here's the part that gets glossed over constantly: GRPO, the RL algorithm behind the DeepSeek-R1 wave, is an outcome method. No value head. No PRM. No per-step critic anywhere in the loop. People say "R1 used RL with reasoning rewards" like that settles the ORM/PRM question. It doesn't. It answers it, and the answer is ORM.
The trick GRPO uses to avoid needing a learned critic is to sample a group. For a given prompt, sample G completions, get their terminal rewards r_1, ..., r_G (correct/incorrect, or whatever your verifier gives back). The baseline isn't a learned value function, it's just the group itself:
r_bar = mean(r_1, ..., r_G)
sigma_r = std(r_1, ..., r_G)
A_i = (r_i - r_bar) / sigma_r
Every token in trajectory i gets advantage A_i. That's the whole critic: "did this sample beat the average of its siblings." No forward pass through a value network, no separate reward model to train and maintain, just arithmetic over a batch you already generated. This is why GRPO was cheap enough to scale the way it did, and it's the direct reason PPO's value head became optional infrastructure instead of load-bearing infrastructure.
But look at what it inherited. A_i is one scalar per trajectory, applied uniformly across every token in that trajectory. Swap r_i for R(y_i) and this is exactly the ORM math from the last section, just with a smarter baseline. GRPO didn't solve the credit-assignment problem, it solved the "how do I get a cheap baseline without training a value model" problem. Different problem. People conflate them because both problems have the word "baseline" in them somewhere.
When G answers to the same prompt all agree (all right, or all wrong), sigma_r goes to zero and every A_i collapses toward zero. No disagreement, no signal. That's a feature, not a bug, but it's a real cost on easy prompts where the model is already consistent, right or wrong.
Here's the tree that makes the whole thing concrete. The model is solving a problem where an early step has a sign error. Two steps later, a second sign error cancels the first one out, and the trajectory arrives at the correct final answer purely by accident. R(y) = 1. The model got lucky.
Under ORM (and under GRPO, since it's ORM with a group baseline), every step in that trajectory, including the sign error at step k, gets advantage (1 - b) if the baseline is below the reward, or A_i > 0 if this sample beat its group. The broken step gets reinforced. You are, in the most literal sense, training the model to repeat a method that only worked because two mistakes canceled. Do this at scale and you're teaching a model to make correlated errors, because correlated errors are the ones that sometimes cancel and get rewarded.
Under a PRM, assuming the step-scorer is any good, V drops right after the flawed step, because the state is worse even though the outcome happened to recover. The TD residual at that step is negative:
delta_k = gamma * V(s_k+1) - V(s_k) < 0
The broken step gets penalized even though the leaf it eventually reaches is green. This is not a subtle distinction. It's the entire reason process supervision exists as a research direction. If you only ever cared about final answers, nobody would have bothered inventing PRMs.
Here's the toy version made concrete in code, a 5-step trajectory with a known-bad step at index 2:
# 5-step trajectory, step 2 (0-indexed) is the flawed one.
# outcome_correct = True (the flawed step's error cancels later)
baseline_b = 0.5
outcome_reward = 1.0 # R(y) = 1, got there by luck
# ORM: same advantage on every step, no matter what happened locally
orm_advantage = [outcome_reward - baseline_b] * 5
print("ORM per-step advantage:", orm_advantage)
# [0.5, 0.5, 0.5, 0.5, 0.5] <- step 2 (the bad one) gets the same boost as the good steps
# PRM: toy value function, drops sharply right after the flawed step
V = [0.6, 0.65, 0.30, 0.55, 0.9, 1.0] # V(s_0) ... V(s_5), s_5 = terminal
gamma = 1.0
prm_advantage = [V[t+1] * gamma - V[t] for t in range(5)]
print("PRM per-step advantage:", prm_advantage)
# [0.05, -0.35, 0.25, 0.35, 0.10] <- step 1 (the flawed step) is negative, everything else positiveSame trajectory. Same correct final answer. Opposite training signal on the flawed step. That flip is the entire post in one printout.
Here's the part nobody wants to say in the same breath as "PRMs give better credit assignment." Denser reward is denser attack surface. If you're scoring every step, every step is now a place the model can learn to game your scorer instead of solving the problem.
A naive PRM often looks like a classifier trained to say "does this look like a good reasoning step." Guess what that classifier picks up on: format. Presence of an equals sign. A cited number. Reasonable length. None of that verifies the step is true, it verifies the step is step-shaped.
def toy_prm_score(step_text: str) -> float:
score = 0.0
if "=" in step_text: score += 0.4
if any(c.isdigit() for c in step_text): score += 0.3
if 20 < len(step_text) < 200: score += 0.3
return score
# a vacuous step that says nothing true, but is well-formatted
print(toy_prm_score("Therefore x = 42, since 7 relates to the problem.")) # 1.0, maxed outThat step is nonsense. It scores perfectly, because a surface-feature PRM can't tell nonsense from math. Train against a scorer like this for long enough and the policy learns to produce plausible-looking filler, not correct reasoning. This is the exact failure mode that made "more reward signal is strictly better" the wrong takeaway from the ORM-vs-PRM comparison. Density helps you learn faster and it hands the model a bigger surface to smooth-talk.
The fix people are converging on for 2026-era reasoning models is grounding each step's reward in something that actually runs: an executable subgoal, a formal lemma checker, a re-derivation of an intermediate quantity that you can compare against ground truth instead of vibes. Verifiable Process Reward Models is exactly this move: score the step by whether its claim checks out, not by whether it looks like a step.
def verifiable_prm_score(step_claim: str, executor) -> float:
result = executor.run(step_claim) # actually evaluates the claim
return 1.0 if result.matches_ground_truth else 0.0
# the vacuous step from before, run for real
print(verifiable_prm_score("Therefore x = 42, since 7 relates to the problem.", executor)) # 0.0Same input string. Different score, because this scorer checks whether the claim is true instead of whether it's shaped like a claim. The cost is real: you now need a verifier for every step, not just the final answer, which is a much heavier build than "did the unit test pass at the end." For math and code this is tractable. For open-ended reasoning it's the hard research problem, not a solved one.
This is the part I actually live in. I run agent fleets at B3, and long tool-use trajectories are the extreme version of this exact fork. A 30-step agent runs, calls a handful of tools, and fails at the end. Which step do you blame?
Pure outcome reward on a 30-step trajectory barely trains. One terminal bit has to explain 30 decisions, most of which had nothing to do with the failure, and your gradient variance is enormous. Naive step-level reward trains fast and gets hacked fast, because the same "does this look like a good step" problem shows up the moment you try to score individual tool calls or intermediate plans instead of the final task outcome.
The answer isn't picking a side. It's grounding your step rewards in something that runs, exactly the verifiable-PRM move, applied to tool use instead of math proofs: did the API call actually return the data the agent claimed, did the file actually get written, did the intermediate state actually match what the agent asserted. That's the same verifiable-reward primitive whether you're scoring a proof step or a tool call. Deterministic checks on probabilistic outputs. Crypto agents live and die by this distinction, and it's why programmable operations, not just programmable money, keeps being the actual bottleneck.
The 1/T-ish variance reduction from dense process rewards is intuition, not a proven rate. Denser signal lowers variance, it's not a theorem with a clean constant in front of it, and I'd distrust anyone who quotes you an exact number here.
Canonical GRPO, as published and as run in the R1 line, is outcome-only. Variants exist that bolt on value heads or step rewards, so "GRPO is an outcome method" describes the standard formulation, not a law of nature.
"PRMs are the most Goodhart-able object in the stack" is a characterization, not a measured ranking against every other reward source you might build. It's the field's working concern, and it's exactly why "verifiable PRM" work exists at all, but treat it as a defensible take, not a benchmark result.
And the lucky-wrong walkthrough assumes the PRM is good. Swap in a bad PRM and it mis-scores the flawed step too, in which case you've traded ORM's honest sparsity for a scorer that's confidently wrong in a new direction. A hackable PRM doesn't fail safe. It fails plausible.
ORM and GRPO are honest and unhackable and they starve on long horizons. PRMs teach faster and catch the lucky-wrong path, and they're the easiest thing you'll ever ship to game. For anything past a handful of steps, the answer isn't "pick a side," it's "ground your step rewards in something that runs." Same primitive, math proof or tool call.
If you build the tree yourself and watch the advantage flip sign on the flawed step when you switch schemes, that's the whole post. Show me.
Sources: