Open-MOPD: Diagnosing and Fixing Capability Imbalance in Multi-Teacher On-Policy Distillation

Gao, Huan-ang; Chi, Haohan; Yan, Yong; Feng, Shiyuan; Wu, Hanlin; Jiang, Zheng; He, Bingxiang; Ma, Wei-Ying; Zhang, Ya-Qin; Zhou, Hao

Open-MOPD Diagnosing and Fixing Capability Imbalance in Multi-Teacher On-Policy Distillation

We introduce the first fully open-source recipe for multi-teacher on-policy distillation.

Open-MOPD reduces the integration gap from 3.50 to 0.31 points in one shared student.

Huan-ang Gao^*,1,2,3, Haohan Chi^*,1,2,3, Yong Yan^1,2,3 Shiyuan Feng^1,2, Hanlin Wu^1,2, Zheng Jiang³, Bingxiang He³ Wei-Ying Ma^1,2, Ya-Qin Zhang^1,2, Hao Zhou^1,2,†

¹SIA-Lab of Tsinghua AIR and ByteDance Seed

²Institute for AI Industry Research (AIR), Tsinghua University

³Department of Computer Science and Technology, Tsinghua University

^*Equal contribution · ^‡Project Lead · ^†Corresponding author

arXiv Code

Models & Data

The first fully open-source multi-teacher on-policy distillation (MOPD) recipe

Open-MOPD specifies the complete path from a public base model to one unified student: mixed-domain supervised fine-tuning, three independently trained domain RL teachers, and hard-routed multi-teacher on-policy distillation, together with the datasets, training settings, checkpoints, and evaluation suite. The recipe balances two constraints. Model capacity must be high enough to close long reasoning trajectories, otherwise truncation confounds the diagnosis; experimental feasibility must remain low-cost enough to repeat the full pipeline and its mechanism ablations. SmolLM3-3B-Base is selected as the operating point between an under-capable 1.7B screen and a capable but substantially more expensive 7B screen.

Preliminary Naive MOPD Results

Naive M-OPD versus RouteOPD: math gap 1.89, code gap 2.45, and IF gap 6.16 points — Integration gap: math 1.89 · code 2.45 · IF 6.16 points.

Math, code, and instruction-following experts can each gain capability through reinforcement learning, yet their strengths do not simply add up when all three teachers update one shared student. Even with correct hard routing by known domain labels, Naive M-OPD remains below the per-domain RouteOPD reference, with the largest deficit on short instruction-following responses.

Diagnosing the Problems

The paper tests routed-teacher disagreement first, then measures the two sources of optimization-budget imbalance: response-token share and the moving teacher–student gap under stale rewards.

Teacher disagreement is measurable, but not the bottleneck

For each generated token, teacher disagreement is the difference between the highest and lowest teacher log-probability assigned to that token. Mean disagreement is 0.126 nat; only 0.62% of tokens exceed 1 nat (3.9% on IF). Masking high-disagreement tokens or replacing their signal with teacher consensus lowers the total score by −0.52 to −0.83.

Routed-teacher disagreement distribution and interventions — Teacher disagreement is measurable, but not the bottleneck.

Prompt share is not token share

Prompt share: 39.8 / 39.8 / 20.3%; gradient-token share: 49.7 / 49.3 / 0.99% (Math / Code / IF). Math/Code responses are ≈ 10,500 tokens; IF is ≈ 409 tokens. IF needs about 33.6× oversampling to get one-third of the token budget.

Prompt share versus gradient-token share, including IF oversampling — Prompt frequency does not determine each domain’s share of gradient tokens.

Moving teacher–student gaps make the budget drift

We estimate a domain’s training budget as its number of response tokens multiplied by the average absolute reward per token. Panel (c) below reports each domain’s percentage of the total budget; because this run gives all three domains the same number of tokens, the percentages are determined only by reward strength. As the teacher–student gaps close at different speeds, IF’s share falls from 48.7% to about 9% within 25 steps, while Code rises from 39.6% to 63.8%.

Teacher-student gap shrinkage causes effective budget drift after token-share balancing — Even after token-share balancing, effective budget drifts as domain gaps shrink at different rates.

Repeated inner updates make rewards stale

Consolidating three domains into one student requires a sufficiently large mixed-domain rollout batch to estimate and balance the Math, Code, and IF token budgets. This large batch is partitioned into multiple PPO minibatch updates. The trajectories are on-policy when sampled, but the student changes after each update while the frozen teacher does not; consequently, a dense reward computed with rollout-time student probabilities becomes inconsistent with the current policy. Rollout-to-current KL reaches 0.059 at K=4 and 0.216 at K=32; the clipped-token fraction reaches 0.86.

Rollout-to-current KL and PPO clipped-token fraction increase with the number of inner updates — As inner updates accumulate, the student moves away from the rollout policy and the cached reward becomes stale.

Method: Open-MOPD

Open-MOPD keeps hard teacher routing and the on-policy distillation objective, but explicitly controls how much update budget each domain receives and refreshes the student-dependent reward before every inner update.

Open-MOPD pipeline: student rollout, hard routing, cached teacher log-probabilities, dense reward, and PPO updates — The teacher scores states visited by the student; it does not generate demonstrations.

Algorithm 1

Open-MOPD training loop

s_dThe fraction of response tokens from domain d in the current batch.

m_dThe mean absolute token reward for domain d—a running measure of the remaining teacher–student gap.

Roll out and route

student rollout → routed teacher scores the student’s top 16 tokens
The student generates the response. Its known Math, Code, or IF label selects one frozen teacher, whose scores are cached once.
Token-share balancing

w_d^share = (1/3) / s_d
If a domain supplies fewer response tokens, each of its tokens receives more weight. The weighted loss therefore assigns one third of its token budget to every domain.
Gap-following allocation

gap factor_d = clip(m_d / mean(m), 0.05, 20) w_d = normalize(w_d^share × gap factor_d)

Domains with more capability left to distill receive more budget. Normalization keeps the total batch-loss scale unchanged.
Reward refresh and PPO update

Refresh token reward r_t^(k) = stopgrad(log p_teacher − log p_student^(k)) × p̃_student^(k)

Apply the Step 3 domain weight r_t^(k) ← w_d(x) × r_t^(k)

Update one shared student PPO update

Before every inner update k, refresh the student-dependent reward, multiply every token by the final weight of its sample’s domain, and only then compute the PPO loss.

Results: 3.50 points → 0.31

We first isolate the contribution of each mechanism, then report detailed results for Math, Code, and Instruction Following, and finally present the complete cross-domain comparison.

Ablation study

Each row adds one mechanism to the setting directly above it. K is the number of PPO minibatch updates per rollout batch.

From Naive M-OPD to the complete Open-MOPD recipe.
Configuration	Share	Gap	Refresh	K	Math	Code	IF	Total	Δ
Naive M-OPD	—	—	—	1	21.26	19.26	43.64	28.05	—
+ token-share	✓	—	—	1	20.55	19.57	47.53	29.22	+1.17
+ gap-following	✓	✓	—	1	21.00	19.31	49.50	29.94	+1.89
Naive matched control	—	—	—	4	21.62	19.72	46.49	29.28	+1.23
+ share + gap	✓	✓	—	4	23.05	21.07	47.16	30.43	+2.38
Open-MOPD	✓	✓	✓	4	22.42	21.73	49.58	31.24	+3.19

Complete results

Total is the mean of the Math, Code, and IF domain averages. Recovery measures recovered RouteRL headroom relative to MixSFT e4.

Complete six-benchmark comparison on SmolLM3-3B.
Method	Math			Code			IF			Total	Recovery
Method	AIME24	AIME25	Avg	LCB v5	LCB v6	Avg	IFEval	IFBench	Avg	Total	Recovery
SmolLM3-3B-Base	2.08	1.72	1.90	3.41	6.17	4.79	16.08	13.00	14.54	7.08	—
MixSFT e4	15.63	20.26	17.95	15.99	19.20	17.60	66.91	16.00	41.46	25.67	—
RouteRL	23.65	24.84	24.24	22.16	21.31	21.73	74.49	27.67	51.08	32.35	100%
RouteOPD	22.34	23.96	23.15	22.28	21.14	21.71	75.60	24.00	49.80	31.55	88.0%
RFT	22.97	23.91	23.44	18.98	19.43	19.21	55.08	18.67	36.87	26.51	12.6%
MixRL	21.15	22.14	21.64	16.59	20.97	18.78	70.24	22.67	46.45	28.96	49.3%
ParamMerge-Avg	18.91	20.99	19.95	18.38	21.20	19.79	70.06	17.67	43.86	27.87	32.9%
ParamMerge-TA	21.93	22.76	22.34	21.74	23.14	22.44	71.53	21.67	46.60	30.46	71.7%
Naive M-OPD	20.92	21.60	21.26	17.53	20.99	19.26	68.61	18.67	43.64	28.05	35.6%
Open-MOPD	21.98	22.86	22.42	20.84	22.63	21.73	74.49	24.67	49.58	31.24	83.4%

Method	Math	Code	IF	Total	Recovery
Base	1.90	4.79	14.54	7.08	—
MixSFT e4	17.95	17.60	41.46	25.67	—
RouteRL	24.24	21.73	51.08	32.35	100%
RouteOPD	23.15	21.71	49.80	31.55	88.0%
RFT	23.44	19.21	36.87	26.51	12.6%
MixRL	21.64	18.78	46.45	28.96	49.3%
Merge-Avg	19.95	19.79	43.86	27.87	32.9%
Merge-TA	22.34	22.44	46.60	30.46	71.7%
Naive M-OPD	21.26	19.26	43.64	28.05	35.6%
Open-MOPD	22.42	21.73	49.58	31.24	83.4%

RouteRL and RouteOPD route each evaluation sample to a separate domain model and are not single deployable students.

Open-MOPD closes the integration gap from 3.50 to 0.31 points and raises RouteRL headroom recovery from 35.6% to 83.4% in one shared student.

BibTeX

@article{gao2026openmopd,
  title={Open-MOPD: Diagnosing and Fixing Capability Imbalance in Multi-Teacher On-Policy Distillation},
  author={Gao, Huan-ang and Chi, Haohan and Yan, Yong and Feng, Shiyuan and Wu, Hanlin and Jiang, Zheng and He, Bingxiang and Ma, Wei-Ying and Zhang, Ya-Qin and Zhou, Hao},
  journal={arXiv preprint},
  year={2026}
}