RL with EmbodiChain

EmbodiChain (image: EmbodiChain).

EmbodiChain is an embodied intelligence lab stack that exposes Gym-style RL tasks. You’ll use RLinf to train an MLP actor-critic with PPO on the EmbodiChain CartPole task.

Overview

Train a state-based MLP policy on EmbodiChain CartPole.

Models

MLP

Algorithms

PPO

Tasks

CartPole

Hardware

1 node · 4 GPUs

You’ll do: install → launch run_embodiment.sh → watch rollout rewards.

Prerequisites: Installation · EmbodiChain package and task resources.

Tasks

Task	Description
CartPole	Balance the pole with state observations from configs/agents/rl/basic/cart_pole/gym_config.json.

Observation and Action

Field	Specification
Observation	A single states tensor built from state_keys: ["qpos", "qvel", "qf"].
Action	2-dim continuous action for policy_setup: cartpole-delta-qpos.
Reward	Task reward from the EmbodiChain Gym config.
Prompt	Not used; this is a low-dimensional state-control recipe.

Installation

First, clone the RLinf repository:

# Mainland China users can use a mirror for faster cloning:
# git clone https://ghfast.top/github.com/RLinf/RLinf.git
git clone https://github.com/RLinf/RLinf.git
cd RLinf

Then set up the dependencies with one of the two methods below — a prebuilt Docker image (recommended) or a custom environment. The general setup (prerequisites, GPU drivers, the in-image switch_env helper, mirrors, and troubleshooting) is documented once in Installation; the commands in this recipe only differ in the Docker image tag and the --env value.

Docker image

docker run -it --rm --gpus all \
   --shm-size 32g \
   --network host \
   --name rlinf \
   -v .:/workspace/RLinf \
   rlinf/rlinf:agentic-rlinf0.3-embodichain

# For mainland China users:
# docker.1ms.run/rlinf/rlinf:agentic-rlinf0.3-embodichain

Switch to the EmbodiChain virtual environment inside the image:

source switch_env embodichain

Custom environment

Install EmbodiChain dependencies:

# Mainland China users can add --use-mirror.
bash requirements/install.sh embodied --env embodichain
source .venv/bin/activate

Warning

EmbodiChain’s dexsim dependency needs libpython3.xx.so. If you hit libpython3.11.so runtime errors with UV’s Python layout, use a Conda environment and rerun bash requirements/install.sh embodied --env embodichain --no-root.

Use the installed package configs by default. To point at a local EmbodiChain checkout, set:

export EMBODICHAIN_PATH=/path/to/EmbodiChain

If a run fails because task resources are missing, download them in the same Python environment:

export EMBODICHAIN_DATA_ROOT=/path/to/data
python -m embodichain.data download --name CartPole
python -m embodichain.data download --name SimResources

Download the Model

No checkpoint is required. The MLP policy starts from scratch.

Run It

Launch the CartPole recipe:

Recipe	Config	Command suffix
MLP + PPO	examples/embodiment/config/embodichain_ppo_cart_pole.yaml	embodichain_ppo_cart_pole

bash examples/embodiment/run_embodiment.sh embodichain_ppo_cart_pole

What this does:

1. Loads the EmbodiChain CartPole Gym JSON through gym_config_path.

2. Creates Ray workers for the actor, rollout, and EmbodiChain env components.

3. Concatenates the configured state fields into states and trains an MLP policy with PPO.

Note

Keep actor.model.obs_dim, actor.model.action_dim, and actor.model.policy_setup aligned with the EmbodiChain task config when you adapt this recipe to another task.

Visualization and Results

The default config logs to W&B. You can switch to TensorBoard by setting:

runner:
  logger:
    logger_backends: ["tensorboard"]

Then launch TensorBoard from the RLinf repo root:

tensorboard --logdir ../results --port 6006

For every logged metric, see Training metrics.

Evaluation and CI

EmbodiChain CartPole is also covered by embodied e2e configs under tests/e2e_tests/embodied/. Set EMBODICHAIN_PATH only when you need a non-default checkout.