RISE-RL
arXiv:2608.09123 · Research Project

RISE-RL

Rubric-Informed Selective Exploration for Open-Ended Reinforcement Learning

Jinkun Hou1*,Zhuo Liu2*,Huimin Ren3†,Hongsheng Xin3,Pan Zhou3,Kun Zhan3
1Peking University · 2Beijing Institute of Technology · 3Li Auto Inc.
* This work was done during an internship at Li Auto Inc.
† Corresponding author
RISE-RL guidance and exploration overview
Guidance expands the reachable policy space. RISE-RL targets repeatedly missed rubric criteria early, then returns to autonomous on-policy exploration.
Motivation

Selective guidance for open-ended alignment

Aligning Large Language Models (LLMs) for open-ended tasks is challenging because responses must satisfy multidimensional criteria without following a single correct generation trajectory. Existing rubric-based reinforcement learning methods compress fine-grained criterion-level feedback into scalar rewards, making persistent capability gaps difficult to target under limited on-policy exploration.

RISE-RL uses repeatedly missed rubric criteria to elicit privileged trajectories that are difficult to discover through unguided exploration alone. It retains only trajectories whose complete-rubric reward exceeds the mean reward of natural rollouts, and re-evaluates them under the original prompt to emphasize behaviors that remain weakly supported by the natural policy.

Target

Criterion-level gaps

Priority is assigned to important rubric criteria that repeatedly fail across natural rollouts.

Filter

Reward-improving guidance

Only privileged trajectories that beat the natural-rollout mean are retained.

Remove

Return to exploration

Guidance is withdrawn after its reward advantage saturates, avoiding permanent dependence.

Method

The RISE-RL framework

RISE-RL augments a natural-prompt GRPO branch with a selective rubric-guidance branch used only during early training. Guided candidates are judged with the complete rubric, filtered by reward gain, and forwarded again under the natural prompt. Policy-support shaping then emphasizes useful tokens that still have low probability under the natural policy.

Full RISE-RL algorithm framework
Figure 2. Natural-prompt GRPO is paired with an early selective-guidance branch. The auxiliary guidance loss is removed after the reward advantage saturates.
Criterion-level selective feedback

Failure frequency and rubric importance jointly determine the highest-priority criteria for each prompt.

Quality-filtered guidance

Guided responses must outperform the mean reward of natural rollouts under the complete original rubric.

Policy-support shaping

Tokens with low support under the natural prompt receive stronger auxiliary weights, focusing learning on hard-to-discover behavior.

Decoupled optimization

The guidance objective is separated from the natural GRPO group and switched off once its marginal benefit plateaus.

Main Results

Strong gains across four open-ended domains

Across Writing, Chat, Health, and Science, RISE-RL achieves the highest average score at both the Qwen3-4B and Qwen3-14B scales. The improvements extend beyond subjective open-ended evaluation to objectively scored medical and scientific benchmarks.

Main results across four open-ended domains
Table 1. Main results across four domains. Green values denote gains over the corresponding initial model; bold rows show RISE-RL.
57.49Qwen3-4B average, +10.25 over the initial model.
70.52Qwen3-14B average, +13.40 over the initial model.
+6.02CreativeWriting-V3 gain at 14B, where standard Rubric-RL is nearly flat.
Training Dynamics

Why guidance should be removed

Rubric-RL, persistent guidance, and RISE-RL comparison
Figure 3. Rubric-RL, persistent guidance, and RISE-RL on Writing and Health. Dashed vertical lines indicate guidance removal.

The comparison reveals a two-stage learning pattern:

  • Rubric-RL: entropy grows slowly, reflecting support-limited unguided exploration.
  • Persistent guidance: entropy remains high and unstable; performance can plateau or degrade when the policy stays dependent on external rubrics.
  • RISE-RL: early guidance expands the policy distribution, then removal consolidates the discovered behavior through autonomous exploration.

After guidance is removed, benchmark performance continues to improve while entropy stabilizes above the Rubric-RL baseline. This supports the central design choice: use guidance to discover behavior, not as a permanent training crutch.

Reference

Citation

@article{hou2026riserl,
  title   = {RISE-RL: Rubric-Informed Selective Exploration for Open-Ended Reinforcement Learning},
  author  = {Hou, Jinkun and Liu, Zhuo and Ren, Huimin and Xin, Hongsheng and Zhou, Pan and Zhan, Kun},
  journal = {arXiv preprint arXiv:2608.09123},
  year    = {2026}
}