PPO training for Math Reasoning

Use this recipe when you want actor-critic PPO for the same math-reasoning task covered by GRPO training for Math Reasoning. PPO and GRPO share most launch and data settings, so this page only lists the PPO-specific differences.

Overview

Use this recipe when you want actor-critic PPO for the same math-reasoning setup used by the GRPO example.

Model

Qwen2.5-1.5B

Algorithm

PPO with GAE advantages and a critic

Data

AReaL-boba math reasoning data

Hardware

Multi-GPU Megatron training

Dataset

We also use the boba dataset. For details, see GRPO training for Math Reasoning.

How PPO Works

Use standard PPO (Proximal Policy Optimization) with GAE advantages and a critic. For the algorithm reference, see PPO.

Run It

1. Config file

For common path, cluster, and runner fields, see Training configuration. Recommended config example:

• examples/reasoning/config/math/qwen2.5-1.5b-ppo-megatron.yaml

2. Launch command

The launch command for PPO training is basically the same as for GRPO training. We also use run_main_grpo_math.sh as the entry script. RLinf automatically determines whether to use PPO training based on whether there are critic-related configurations in the YAML config file and the value of adv_type (PPO typically uses gae as the advantage function).

Visualization and Results

We fine-tune the Qwen2.5-1.5B-Instruct model with PPO. The orange line is RLinf and the blue line is VeRL; both use the same algorithm configuration. For common metric meanings, see Training metrics.

Since the base capability of the Qwen2.5-1.5B-Instruct model is relatively weak, the overall reward values are low. However, as training progresses, the reward values increase significantly.

MATH 1.5B PPO