RLinf Documentation

Welcome to RLinf!

RLinf is a flexible and scalable open-source infrastructure designed for post-training foundation models via reinforcement learning. The ‘inf’ in RLinf stands for Infrastructure, highlighting its role as a robust backbone for next-generation training. It also stands for Infinite, symbolizing the system’s support for open-ended learning, continuous generalization, and limitless possibilities in intelligence development.

---

---

RLinf is unique with:

• Macro-to-Micro Flow: a new paradigm M2Flow, which executes macro-level logical flows through micro-level execution flows, decoupling logical workflow construction (programmable) from physical communication and scheduling (efficiency).

• Flexible Execution Modes

  • Collocated mode: shares all GPUs across all workers.

  • Disaggregated mode: enables fine-grained pipelining.

  • Hybrid mode: a customizable combination of different placement modes, integrating both collocated and disaggregated modes.

• Auto Scheduling

  • Dynamic Scheduling: dynamically schedule resource allocation, maximizing resource utilization.

  • Static Scheduling: automatically select the most suitable execution mode based on the training workload, without the need for manual resource allocation.

• Embodied Agent Support

  • Fast adaptation support for mainstream VLA models: OpenVLA, OpenVLA-OFT, π₀, GR00T-N1.5

  • Support for mainstream CPU & GPU-based simulators via standardized RL interfaces: ManiSkill3, LIBERO, IsaacLab

  • Enabling the first RL fine-tuning of the π₀ model family with a flow-matching action expert.

RLinf is fast with:

• Hybrid mode with fine-grained pipelining: achieves a 120%+ throughput improvement compared to other frameworks.

• Automatic Online Scaling Strategy: dynamically scales training resources, with GPU switching completed within seconds, further improving efficiency by 20–40% while preserving the on-policy nature of RL algorithms.

RLinf is flexible and easy to use with:

• Multiple Backend Integrations

  • FSDP + Hugging Face: rapid adaptation to new models and algorithms, ideal for beginners and fast prototyping.

  • Megatron + SGLang: optimized for large-scale training, delivering maximum efficiency for expert users with demanding workloads.

• Adaptive communication via the asynchronous communication channel

• Built-in support for popular RL methods, including PPO , GRPO , DAPO , Reinforce++ , and more.

---

• Quickstart
• SOTA RL Training Reproduction
  • Installation
  • Quickstart 1: PPO Training of VLAs on Maniskill3
  • Quickstart 2: GRPO Training of LLMs on MATH
  • Multi-node Training
  • Evaluation Tutorial 1: Embodied VLA
  • Evaluation Tutorial 2: Math Reasoning LLM

---

• Tutorials
  • Configuration
    • Basic Configuration
    • Embodiment Configuration
    • Agentic RL Configuration
    • Heterogenous Hardware Setup
    • Checkpoint Resume
    • Training Visualisation
  • Usage and Programming Tutorial
    • Worker-Based Programming Interface
    • Worker Placement Strategy
    • High-Level Programming Flow
    • Use Channel for Communication
    • Checkpoint Convertor
    • Multi-Node Ray Cluster Setup
    • Flexible Execution Modes
  • Embodied Intelligence
    • Simulators, Robots, and Models
    • Real-World Robot Training Launch
    • Cloud-Edge Training Setup
    • Replay Buffer Tutorial
    • Data Collection
    • Reward Model Guide
  • Agentic RL
  • Supported RL Algorithms
    • Proximal Policy Optimization (PPO)
    • Group Relative Policy Optimization (GRPO)
    • Decoupled Clip and Dynamic sAmpling Policy Optimization (DAPO)
    • REINFORCE++
    • Soft Actor-Critic (SAC) Algorithm
    • Cross-Q
    • Reinforcement Learning with Prior Data (RLPD)
    • Implicit Q-Learning (IQL) Algorithm
    • Async Proximal Policy Optimization (Async PPO)
  • Extending the Framework
    • Adding New Environment
    • Adding New Models with FSDP+HuggingFace
    • Adding New Models with Megatron+SGLang
    • Add a New Model for SFT Training with FSDP + HuggingFace
    • Megatron-Bridge
    • Weight Synchronization
  • Advanced Features
    • LoRA Integration
    • 5D Parallelism Configuration
    • Ray-Based Cluster Launching
    • Adaptive P2P Communication
    • Switch SGLang Versions
    • GPU Profiling
    • Dynamic Scheduling
    • Auto Placement
  • Release Notes

---

• Example Gallery
  • Embodied Scenarios
  • Agentic Scenarios
  • System-level Optimizations

---

• Blog
  • Comparison with VeRL
  • A First Look at the “Last Mile” of Agent Deployment: Cursor Online Reinforcement Learning
  • Accelerating the “ImageNet Moment” of Embodied AI: RLinf Brings a 25× System Optimization to BEHAVIOR

---

• Publications
  • RLinf-USER: Unified System for Real-world Online Policy Learning
  • RLinf-VLA: A Unified and Efficient Framework for VLA+RL Training
  • Beyond Imitation: Reinforcement Learning-Based Sim-Real Co-Training for VLA Models
  • RLinf: Flexible and Efficient Large-scale Reinforcement Learning via Macro-to-Micro Flow Transformation
  • πRL: Online RL Fine-tuning for Flow-based Vision-Language-Action Models
  • WoVR: World Models as Reliable Simulators for Post-Training VLA Policies with RL
  • WideSeek-R1: Exploring Width Scaling for Broad Information Seeking via Multi-Agent Reinforcement Learning

---

• APIs
  • Worker Interface
  • Placement Interface
  • Cluster Interface
  • Channel Interface
  • Actor Interface
  • Rollout Interface
  • Environment Interface
  • Data Interface
  • Embodied Data Interface
  • Replay Buffer

---

• FAQs