SIMPLE Evaluation#
Evaluate Psi0 on eight SIMPLE Teleop tasks through RLinf. Each task uses its matching checkpoint and reset states. For PPO post-training, see the Psi0 guide.
Scope#
The current scope is the eight Teleop tasks with task-specific checkpoints in
the pinned Psi0 release. G1WholebodyBendPickMP-v0 and
G1WholebodyTabletopGraspMP-v0 use different state, action, and System-0
protocols and are not supported yet.
Set Up the Environment#
All eight tasks share steps 1–4 of the Psi0 guide. First, install and activate the runtime from the RLinf repository root. To use Docker instead, see step 1.
# Add --use-mirror for faster downloads in mainland China.
bash requirements/install.sh embodied --model psi0 --env simple
source .venv/bin/activate
Then complete steps 2–4:
Download the simulation assets, or reuse assets you already have.
Link the assets to
${VIRTUAL_ENV}/SIMPLE/data.
Note
The Psi0 loader needs the raw Hugging Face snapshot of the Qwen metadata. Set HF_HOME when downloading it instead of using --local-dir, and export the same HF_HOME before evaluation.
Step 5 depends on the task. See Task-Specific Assets.
Select a Task and Download Assets#
The table lists the eight valid Teleop substitutions in the
pinned Psi0 checkpoint release.
Official result is the Psi0 success count over 10 episodes at each of
levels 0, 1, and 2 in the
pinned SIMPLE results.
|
Run name |
|
Horizon |
Official result |
|---|---|---|---|---|
|
|
|
800 |
10/10/6 |
|
|
|
800 |
7/7/10 |
|
|
|
1200 |
7/5/6 |
|
|
|
800 |
10/9/9 |
|
|
|
1000 |
10/10/10 |
|
|
|
1000 |
7/5/4 |
|
|
|
1000 |
3/3/4 |
|
|
|
1000 |
7/6/3 |
Task-Specific Assets#
The commands below use OpenOven. All tasks share the artifact root
PSI0_ARTIFACT_ROOT, so set it once. When switching tasks, change only the
annotated SIMPLE_TASK, PSI0_RUN_REL, and SIMPLE_SCENE. All three
values must come from the same table row. The remaining commands derive the
matching paths from these values. The HSSD scene is unpacked into
SIMPLE_ASSETS_DIR from step 3.
# Set once. Reuse for every SIMPLE task.
export PSI0_ARTIFACT_ROOT=/path/to/psi0-simple-artifacts
# Replace these three values together for each task.
export SIMPLE_TASK=G1WholebodyOpenOvenTeleop-v0 # Replace the task ID.
# Replace only the run name after psi0/simple-checkpoints/.
export PSI0_RUN_REL=psi0/simple-checkpoints/g1wholebodyopenoventeleop-v0.simple.flow1000.cosine.lr1.0e-04.b128.gpus8.2605120604
export SIMPLE_SCENE=102344280 # Replace the HSSD scene ID.
mkdir -p "${PSI0_ARTIFACT_ROOT}"
hf download USC-PSI-Lab/psi-model \
"${PSI0_RUN_REL}/argv.txt" \
"${PSI0_RUN_REL}/run_config.json" \
"${PSI0_RUN_REL}/checkpoints/ckpt_40000/model.safetensors" \
--revision d34a91932d25c45ef211582315b9224c7dc8ace9 \
--local-dir "${PSI0_ARTIFACT_ROOT}/psi-model"
hf download USC-PSI-Lab/psi-data "simple-eval/${SIMPLE_TASK}.zip" \
--repo-type dataset \
--revision 6eeff2d02fdaac5dd4f3e84244fc83d3fad2c203 \
--local-dir "${PSI0_ARTIFACT_ROOT}/psi-data"
export PSI0_RUN_DIR="${PSI0_ARTIFACT_ROOT}/psi-model/${PSI0_RUN_REL}"
export SIMPLE_RESET_DIR="${PSI0_ARTIFACT_ROOT}/reset-states/${SIMPLE_TASK}"
mkdir -p "${SIMPLE_RESET_DIR}"
unzip -q "${PSI0_ARTIFACT_ROOT}/psi-data/simple-eval/${SIMPLE_TASK}.zip" \
-d "${SIMPLE_RESET_DIR}"
hf download USC-PSI-Lab/SIMPLE "scenes_hssd_${SIMPLE_SCENE}.zip" \
--repo-type dataset \
--local-dir "${SIMPLE_ASSETS_DIR}"
unzip -o -q "${SIMPLE_ASSETS_DIR}/scenes_hssd_${SIMPLE_SCENE}.zip" -d "${SIMPLE_ASSETS_DIR}"
rm "${SIMPLE_ASSETS_DIR}/scenes_hssd_${SIMPLE_SCENE}.zip"
Run Evaluation#
evaluations/simple/simple_psi0_eval.yaml defaults to OpenOven, one episode,
and DR0. Set the local asset paths to run it:
bash evaluations/run_eval.sh simple simple_psi0_eval \
rollout.model.model_path="${PSI0_RUN_DIR}" \
env.eval.reset_dataset.path="${SIMPLE_RESET_DIR}"
Evaluation reports eval/success; logs and videos are saved under
logs/<timestamp>-simple_psi0_eval/.
For a full evaluation, run 10 episodes at each of the three DR levels:
for SIMPLE_DR_LEVEL in 0 1 2; do
bash evaluations/run_eval.sh simple simple_psi0_eval \
rollout.model.model_path="${PSI0_RUN_DIR}" \
env.eval.reset_dataset.path="${SIMPLE_RESET_DIR}" \
env.eval.reset_dataset.dr_level="${SIMPLE_DR_LEVEL}" \
env.eval.reset_dataset.num_episodes=10 \
env.eval.rollout_epoch=10
done
Switch to Another Task#
Reuse the same config for another Teleop task. For XMovePick, update all five variables; exported paths do not change automatically:
export SIMPLE_TASK=G1WholebodyXMovePickTeleop-v0
export PSI0_RUN_REL=psi0/simple-checkpoints/g1wholebodyxmovepick-v0.simple.flow1000.cosine.lr1.0e-04.b128.gpus8.2604022205
export SIMPLE_SCENE=107734119_175999932
export PSI0_RUN_DIR="${PSI0_ARTIFACT_ROOT}/psi-model/${PSI0_RUN_REL}"
export SIMPLE_RESET_DIR="${PSI0_ARTIFACT_ROOT}/reset-states/${SIMPLE_TASK}"
If these assets are not ready, rerun the model, reset-data, and scene download commands with the new variables. Then evaluate:
bash evaluations/run_eval.sh simple simple_psi0_eval \
rollout.model.model_path="${PSI0_RUN_DIR}" \
env.eval.reset_dataset.path="${SIMPLE_RESET_DIR}" \
env.eval.init_params.task_id="simple/${SIMPLE_TASK}" \
env.eval.max_episode_steps=800 \
env.eval.max_steps_per_rollout_epoch=816
The checkpoint, reset data, and task_id must match. In the three-level loop,
also add env.eval.init_params.task_id="simple/${SIMPLE_TASK}" and keep both
asset-path and step-limit overrides.
max_steps_per_rollout_epoch is the smallest multiple of 24 that is not
smaller than the task horizon.
Task |
|
|
|---|---|---|
XMovePick, Handover, XMoveBendPick |
800 |
816 |
CloseDoor, OpenOven, OpenFaucet, PickAndPlaceAndHugContainer |
1000 |
1008 |
LocomotionPickBetweenTables |
1200 |
1200 |
All eight tasks use mujoco_isaac, 50 Hz rendering, 200 Hz physics, at most
300 stabilization steps, and the 30/24/6 RTC protocol. The official result has
only 10 episodes per level. Report each level separately; neither a one-episode
smoke nor an aggregate count establishes strict reproduction.
Evaluate a Trained Checkpoint#
Use the original run directory to build the model and processor, then load the
saved PPO weights with runner.ckpt_path. Set
rollout.model.add_value_head=true to restore the saved value-head parameters.
For an OpenOven checkpoint, select the matching assets and evaluate at the official 1,000-step horizon:
export SIMPLE_TASK=G1WholebodyOpenOvenTeleop-v0
export PSI0_RUN_REL=psi0/simple-checkpoints/g1wholebodyopenoventeleop-v0.simple.flow1000.cosine.lr1.0e-04.b128.gpus8.2605120604
export PSI0_RUN_DIR="${PSI0_ARTIFACT_ROOT}/psi-model/${PSI0_RUN_REL}"
export SIMPLE_RESET_DIR="${PSI0_ARTIFACT_ROOT}/reset-states/${SIMPLE_TASK}"
export PSI0_PPO_CHECKPOINT=/absolute/path/to/global_step_20
bash evaluations/run_eval.sh simple simple_psi0_eval \
rollout.model.model_path="${PSI0_RUN_DIR}" \
rollout.model.add_value_head=true \
runner.ckpt_path="${PSI0_PPO_CHECKPOINT}/actor/model_state_dict/full_weights.pt" \
env.eval.reset_dataset.path="${SIMPLE_RESET_DIR}" \
env.eval.reset_dataset.num_episodes=10 \
env.eval.rollout_epoch=10