More Works from Our Lab

Paper Title 1

Brief description of the work and its main contribution.

Conference/Journal 2024
Paper Title 2

Brief description of the work and its main contribution.

Conference/Journal 2023
Paper Title 3

Brief description of the work and its main contribution.

Conference/Journal 2023

LIBERO-PRO: Towards Robust and Fair Evaluation of Vision-Language-Action Models Beyond Memorization

Xueyang Zhou1, Yangming Xu1, Guiyao Tie1, Yongchao Chen2,3, Guowen Zhang1, Duanfeng Chu4, Pan Zhou1, Lichao Sun5
1Huazhong University of Science and Technology, 2Harvard University, 3Massachusetts Institute of Technology, 4Wuhan University of Technology, 5Lehigh University
Paper Code arXiv
LIBERO-PRO Teaser
Figure 1. Overview of LIBERO-PRO: An evaluation extension for embodied VLA benchmarks.

LIBERO-PRO introduces more challenging and fair evaluation environments to drive VLA models toward true generalization and comprehension.

Abstract

LIBERO has emerged as a widely adopted benchmark for evaluating Vision-Language-Action (VLA) models; however, its current training and evaluation settings are problematic, often leading to inflated performance estimates and preventing fair model comparison. To address these issues, we introduce LIBERO-PRO, an extended LIBERO benchmark that systematically evaluates model performance under reasonable perturbations across four dimensions: manipulated objects, initial states, task instructions, and environments. Experimental results reveal that, although existing models achieve over 90% accuracy under the standard LIBERO evaluation, their performance collapses to 0.0% under our generalized setting. Crucially, this discrepancy exposes the models’ reliance on rote memorization of action sequences and environment layouts from the training set, rather than genuine task understanding or environmental perception. For instance, models persist in executing grasping actions when the target object is replaced with irrelevant items, and their outputs remain unchanged even when given corrupted instructions or even messy tokens. These findings expose the severe flaws in current evaluation practices, and we call on the community to abandon misleading methodologies in favor of robust assessments of model generalization and comprehension.

The VLA’s High Scores Reflect Rote Memorization, Not The Acquisition Of Effective Policy

Finding 1: Does the Model Generalize to New Objects? VLA completely don't know to grasp what is object

LIBERO Original Task Original Task: Pick the salad dressing and place it in the basket. ✔
Same Object Change the color of the salad dressing to red. ✔
Other Object Replace the salad dressing with alphabet soup. ✖
Other Object Replace the salad dressing with alphabet soup. ✖

The salad dressing on the workspace was replaced with alphabet soup. The VLA still grasped it as if it were salad dressing, demonstrating that it follows a fixed action trajectory rather than understanding the object.

Finding 2: Does the Model Generalize to Varied Instructions? VLA completely don't know what instruction

LIBERO Original Task Original Task: Pick the salad dressing and place it in the basket. ✔
Semantic Perturbation Grab salad dressing and set it in basket. ✔
Task Change Pick the tomato sauce and place it in the basket. ✖
Meaningless instruction xxx. ✖

Finding 3: How Sensitive is the Model to Object Placement? VLA can hardly handle changes in the position of objects

LIBERO Original Task Pick the alphabet soup and place it in the basket.✔
Position Perturbation TaskX-axis +0.1✔
Position Perturbation TaskX-axis +0.2✖
Position Perturbation TaskX-axis +0.3✖
Position Perturbation TaskX-axis +0.4✖
Position Perturbation TaskX-axis +0.5✖
LIBERO Original TaskPick the alphabet soup and place it in the basket.✔
Position Perturbation TaskY-axis +0.1✖
Position Perturbation TaskY-axis +0.2✖
Position Perturbation TaskY-axis +0.3✖
Position Perturbation TaskY-axis +0.4✖
Position Perturbation TaskY-axis +0.5✖

Finding 4: Is the Model Truly Understanding Tasks?

Task 1Put the bowl on top of the cabinet.✔
Task 2Turn on the stove.✔
Combined TaskPut the bowl on cabinet then turn on stove.✖

Quantitative Experimental Analysis

Case Study

OpenVLA in LIBERO-GOAL

LIBERO Original Task Original Task: Open the middle layer of the drawer. ✔
LIBERO-PRO Task Object Perturbation: Change the cabinet to yellow. ✔
LIBERO-PRO Task Initial Position Perturbation: Change the initial position of the cabinet. ✖
LIBERO-PRO Task Semantic Perturbation: Open the bottom layer of the drawer. ✔
LIBERO-PRO Task Task Perturbation: Open the top layer of the drawer. ✖

Pi0.5 in LIBERO-SPATIAL

LIBERO Original Task Original Task: Pick the akita black bowl on the ramekin and place it on the plate. ✔
LIBERO-PRO Task Object Perturbation: Change the plate to yellow. ✔
LIBERO-PRO Task Initial Position Perturbation: Change the initial position of the plate. ✔
LIBERO-PRO Task Semantic Perturbation: Grab the akita black bowl upon the ramekin and place it on the plate. ✔
LIBERO-PRO Task Task Perturbation: Pick the akita black bowl on the cookie box and place it on the plate. ✖

Pi0 in LIBERO-10

LIBERO Original Task Original Task: Put the yellow and white mug in the microwave and close it. ✔
LIBERO-PRO Task Object Perturbation: Change the other cup to yellow and white as well. ✔
LIBERO-PRO Task Initial Position Perturbation: Swap the positions of the two cups. ✖
LIBERO-PRO Task Semantic Perturbation: Grab the yellow and white mug, put it in the microwave and close it. ✔
LIBERO-PRO Task Task Perturbation: put the white mug in the microwave and close it. ✖

OpenVLA in LIBERO-OBJECT

LIBERO Original Task Original Task: Pick the alphabet soup and place it in the basket. ✔
LIBERO-PRO Task Object Perturbation: Change the color of alphabet soup. ✔
LIBERO-PRO Task Initial Position Perturbation: Change the initial position of alphabet soup. ✖
LIBERO-PRO Task Semantic Perturbation: Take the alphabet soup and put it in the basket. ✔
LIBERO-PRO Task Task Perturbation: Pick the cream cheese and place it in the basket. ✖

BibTeX

@article{Liberopro2025,
  title={LIBERO-PRO: Towards Robust and Fair Evaluation of Vision-Language-Action Models Beyond Memorization},
  author={Xueyang Zhou and Yangming Xu and Guiyao Tie and Yongchao Chen and Guowen Zhang and Duanfeng Chu and Pan Zhou and Lichao Sun},
  journal={[arXiv preprint arXiv:2510.03827]},
  year={2025},
  url={https://arxiv.org/abs/2510.03827}
}