AgentLightning RL Training (calc_x)#

calc_x is an AgentLightning example in RLinf for training a math-solving agent. The agent reads a question, produces reasoning and an answer, and then receives feedback for RL updates.

Overview#

Use this recipe to train a calculator-backed math agent with Agent Lightning and RLinf’s distributed trainer.

Model

Qwen2.5-1.5B-Instruct

Algorithm

Multi-turn agent RL

Tools

MCP calculator and AutoGen agent chat

Hardware

One node with at least one 40 GB GPU

Installation#

AgentLightning 0.3.0 requires the OpenTelemetry versions constrained by RLinf because OpenTelemetry 1.45 changed the exporter implementation used to route rollout spans to the Lightning Store. After creating the base environment, install the compatible OpenTelemetry set before the dependencies for this example:

cd /path/to/RLinf
bash requirements/install.sh agentic
source .venv/bin/activate
uv pip install -r requirements/agentic/agentlightning_constraints.txt
uv pip install "agentlightning==0.3.0" "litellm<1.95" \
  "autogen-agentchat" "autogen-ext[openai]" "mcp>=1.10.0" \
  "mcp-server-calculator" "openai==2.32.0"

For other installation options and supported platforms, see RLinf Installation.

Data Preparation#

Download and extract the calc_x dataset (Google Drive). See the download link here.

Run It#

Go to the example directory:

cd /path/to/RLinf/examples/agent/agentlightning/calc_x

Choose the config you want to run, then set the model and dataset paths in the same config file. For example, the training command below uses config/qwen2.5-1.5b-enginehttp-multiturn.yaml:

rollout:
  model:
    model_path: /path/to/model/Qwen2.5-1.5B-Instruct

data:
  train_data_paths: ["/path/to/train.parquet"]
  val_data_paths: ["/path/to/test.parquet"]

Start training:

bash run_calc_x.sh qwen2.5-1.5b-enginehttp-multiturn

You can also keep the config file unchanged and pass Hydra overrides on the command line:

bash run_calc_x.sh qwen2.5-1.5b-enginehttp-multiturn \
  rollout.model.model_path=/path/to/Qwen2.5-1.5B-Instruct \
  data.train_data_paths='["/path/to/train.parquet"]' \
  data.val_data_paths='["/path/to/test.parquet"]'

To train with trajectory-level advantages instead, use the matching trajectory config and set the same paths there or via overrides:

bash run_calc_x.sh qwen2.5-1.5b-enginehttp-trajectory

Visualization and Results#

Example training / metric curves from a calc_x run (logged metrics may vary by config and seed):

AgentLightning calc_x training curves — AgentLightning `calc_x` training curves#

Standalone Evaluation#

For HF evaluation, set rollout.model.model_path in the matching *_eval.yaml. Examples:

bash run_calc_x.sh qwen2.5-1.5b-enginehttp-multiturn_eval
bash run_calc_x.sh qwen2.5-1.5b-enginehttp-trajectory_eval