Direct-OPD
中文

Weak-to-Strong Generalization via Direct On-Policy Distillation

Strong student models can learn from small RL teachers: Direct-OPD transfers the RL-induced policy shift as dense supervision on the student's own on-policy states, instead of imitating the weaker teacher's final policy.

Shiyuan Feng1,2,4,*, Huan-ang Gao1,2,3,*,‡, Haohan Chi1,2,3,*, Hanlin Wu1,2, Zhilong Zhang1,2, Zheng Jiang3, Bingxiang He3 Wei-Ying Ma1,2, Ya-Qin Zhang1,2, Hao Zhou1,2,†

*Equal contribution. ‡Project Lead. †Corresponding author.

1SIA-Lab of Tsinghua AIR and ByteDance Seed

2Institute for AI Industry Research (AIR), Tsinghua University

3Department of Computer Science and Technology, Tsinghua University

4Peking University

arXiv Code Models
Direct-OPD intro comparison

RLVR has already shown that reinforcement learning can improve reasoning models, but scaling it to every larger model means repeating exploration under sparse outcome rewards. Direct-OPD reuses exploration that a small model has already completed: first let the small model learn a useful direction through RL, then apply that direction to the stronger model's own on-policy states. In contrast, vanilla OPD distills the post-RL teacher distribution itself, which can pull a stronger student back toward the weaker teacher instead of transferring only the RL-induced reward direction.

Method: transfer the policy shift, not the teacher itself

Policy-as-reward for the small teacher

Apply the KL-regularized RL identity to the small teacher. If the teacher is trained from $\pi_{\mathrm{ref,teacher}}$ with external reward $R_{\mathrm{teacher}}$ and KL coefficient $\beta_T$, its optimum satisfies:

$$\pi^*_{\mathrm{teacher}}(y\mid x)\propto \pi_{\mathrm{ref,teacher}}(y\mid x)\exp(R_{\mathrm{teacher}}(x,y)/\beta_T).$$

Rearranging gives:

$$\log\frac{\pi^*_{\mathrm{teacher}}(y\mid x)} {\pi_{\mathrm{ref,teacher}}(y\mid x)} =\frac{1}{\beta_T}R_{\mathrm{teacher}}(x,y)-\log Z_T(x).$$

Since $\log Z_T(x)$ is constant for all responses to the same prompt, the teacher/reference log-ratio recovers the teacher's RL reward up to a positive scale and a prompt-level constant.

The teacher pair exposes that reward

In practice, the post-RL checkpoint $\pi_{\mathrm{teacher}}$ is the teacher produced by that RL run. Comparing it to the pre-RL checkpoint $\pi_{\mathrm{ref,teacher}}$ therefore exposes the reward-like signal written into the teacher by external RL supervision:

$$\Delta_T(y\mid x):= \log\frac{\pi_{\mathrm{teacher}}(y\mid x)} {\pi_{\mathrm{ref,teacher}}(y\mid x)} =\frac{1}{\beta_T}R_{\mathrm{teacher}}(x,y)-\log Z_T(x).$$

This is the object Direct-OPD transfers: the teacher-derived implicit reward, not the teacher policy $\pi_{\mathrm{teacher}}$ itself.

Direct-OPD transforms it into student supervision

Because language-model policies factorize over prefixes, the sequence-level shift decomposes into token-level rewards:

$$R_{\mathrm{student},t}(v\mid s_t)\leftarrow \Delta_{T,t}(v\mid s_t)= \log\frac{\pi_{\mathrm{teacher}}(v\mid s_t)} {\pi_{\mathrm{ref,teacher}}(v\mid s_t)}.$$

During Direct-OPD, the student samples its own rollouts, keeps the top-k actions it actually considers at each visited prefix, and scores those actions with $R_{\mathrm{student},t}$. The strong student is therefore trained by the implicit dense external supervision recovered from the small model's RL process, rather than by imitating the weaker teacher or rerunning sparse-reward RL on the target model.

Experiment 1: directions learned by small-model RL improve stronger students

Direct-OPD improves multiple student models. The teacher does not need to be stronger than the student; what gets reused is the direction formed by RL.

(a) JustRL policy-shift transfer

Model AIME24 AIME25
Teacher ref 28.5 24.0
Teacher RL 51.3 37.5
Qwen3-1.7B 48.3 36.8
+ Direct-OPD 62.4 +14.1 46.3 +9.5
Qwen3-4B 72.5 65.6
+ Direct-OPD 77.6 +5.1 68.8 +3.2
R1-Distill-7B 56.7 40.5
+ Direct-OPD 63.1 +6.4 48.8 +8.3

(b) QuestA policy-shift transfer

Model AIME24 AIME25
Teacher ref 61.8 49.5
Teacher RL 72.5 62.3
Qwen3-1.7B 49.1 36.8
+ Direct-OPD 59.0 +9.9 43.1 +6.3
R1-Distill-7B 56.3 39.5
+ Direct-OPD 61.2 +4.9 44.0 +4.5
JustRL policy-shift transfer results
The policy shift from JustRL-1.5B improves Qwen3-1.7B, Qwen3-4B, and R1-Distill-7B. The latter two students already start above the post-RL teacher, suggesting that the gain comes from the RL direction rather than the weak teacher distribution itself.

Experiment 2: find the direction on a small model, then transfer it to a larger model

Running RL directly on a larger model asks that model to rediscover credit assignment through many rollouts. Direct-OPD moves this exploration to a small model, then transfers the resulting policy shift to the larger model.

Weak-to-strong compute comparison
Takeaway. Compared with direct large-model RL, running RL on a small model first and then transferring to the large model has two advantages: small-model RL is more stable, easier, and cheaper; the following Direct-OPD stage only needs short training to move that advantage to the larger model.

Experiment 3: directions learned by different RL processes can be composed sequentially

Direct-OPD transfers directions rather than the full post-RL teacher distribution. Policy shifts learned by different teacher pairs can therefore be applied sequentially to the same student.

Sequential policy-shift composition
Qwen3-1.7B first receives the JustRL signal and then the QuestA signal, improving on AIME 2024 from 48.3 to 63.8.
Takeaway. Different RL training processes learn different capabilities. Direct-OPD can compose those capabilities as policy shifts inside the same student model.
BibTeX
@misc{feng2026directopd,
  title  = {Weak-to-Strong Generalization via Direct On-Policy Distillation},
  author = {Shiyuan Feng and Huan-ang Gao and Haohan Chi and Hanlin Wu and Zhilong Zhang and Zheng Jiang and Bingxiang He and Wei-Ying Ma and Ya-Qin Zhang and Hao Zhou},
  year   = {2026},
  eprint = {2607.05394},
  archivePrefix = {arXiv},
  primaryClass = {cs.LG},
  url    = {https://arxiv.org/abs/2607.05394}
}