Chaoyi Pan

Chaoyi Pan

PhD Student, Carnegie Mellon University

I am a 4th-year PhD student in Electrical and Computer Engineering affiliated with the Robotics Institute at Carnegie Mellon University, working with Prof. Guanya Shi (LeCAR Lab) and Prof. Guannan Qu.

My research interests lie at the intersection of physics-based robot data generation and the understanding of generative models in control.

News

Publications

* denotes equal contribution; † denotes equal advising.

Action Representation in Generative Control

Much Ado About Noising teaser

Much Ado About Noising: Dispelling the Myths of Generative Robotic Control

Chaoyi Pan, Giri Anantharaman, Nai-Chieh Huang, Claire Jin, Daniel Pfrommer, Chenyang Yuan, Frank Permenter, Guannan Qu, Nicholas Boffi, Guanya Shi, Max Simchowitz

ICLR 2026

TL;DR: Generative control policies benefit from supervised iterative refinement and suitable stochasticity. A simple two-step regression policy matches flow-based policies on behavior-cloning benchmarks, challenging explanations centered on multimodal action distributions.

Physics-based Data Generation

SPIDER teaser

SPIDER: Scalable Physics-Informed Dexterous Retargeting

Chaoyi Pan, Changhao Wang, Haozhi Qi, Zixi Liu, Homanga Bharadhwaj, Akash Sharma, Tingfan Wu, Guanya Shi†, Jitendra Malik, Francois Hogan

IROS 2026

TL;DR: SPIDER turns human motion demonstrations into physically feasible robot trajectories through physics-based sampling and contact guidance, scaling data generation across humanoids and dexterous hands.

Model-based Diffusion teaser

Model-Based Diffusion for Trajectory Optimization

Chaoyi Pan*, Zeji Yi*, Guanya Shi†, Guannan Qu†

NeurIPS 2024

TL;DR: MBD uses known dynamics to compute diffusion scores and optimize trajectories without training data, while optionally incorporating imperfect demonstrations to guide solutions for contact-rich tasks.

Sampling-based Optimization

Hybrid Feedback Sampling for Sample-Efficient Model Predictive Control overview

Hybrid Feedback Sampling for Sample-Efficient Model Predictive Control

Chaoyi Pan, Zeji Yi, John Zhang, Zachary Manchester, Guannan Qu, Guanya Shi

arXiv 2026

TL;DR: FS-MPC combines feedback-guided sampling with local and global search to improve sample efficiency and control unstable, contact-rich systems, including real humanoid locomotion and manipulation.

Global Convergence of Sampling-Based Nonconvex Optimization through Diffusion-Style Smoothing overview

Global Convergence of Sampling-Based Nonconvex Optimization through Diffusion-Style Smoothing

Zeji Yi*, Chaoyi Pan*, Guanya Shi, Guannan Qu

TMLR 2026

TL;DR: We analyze sampling-based optimization through diffusion-style smoothing and develop DIDA, an annealed algorithm with a global convergence guarantee under the stated assumptions.

DIAL-MPC teaser

Full-Order Sampling-Based MPC for Torque-Level Locomotion Control via Diffusion-Style Annealing

Haoru Xue*, Chaoyi Pan*, Zeji Yi, Guannan Qu, Guanya Shi

ICRA 2025 Best Paper Finalist

TL;DR: DIAL-MPC uses diffusion-style annealing to refine sampled control trajectories, enabling real-time, training-free torque control with full quadruped dynamics, including precise real-world jumping with payloads.

CoVO-MPC teaser

CoVO-MPC: Theoretical Analysis of Sampling-Based MPC and Optimal Covariance Design

Zeji Yi*, Chaoyi Pan*, Guanqi He, Guannan Qu†, Guanya Shi†

L4DC 2024

TL;DR: We analyze MPPI convergence and use the results to design an optimal sampling covariance schedule, improving sampling-based control in simulation and real-world agile quadrotor tasks.

Robotics Systems

Learning Gentle Humanoid Locomotion and End-Effector Stabilization Control overview

Learning Gentle Humanoid Locomotion and End-Effector Stabilization Control

Yitang Li, Yuanhang Zhang, Wenli Xiao, Chaoyi Pan, Haoyang Weng, Guanqi He, Tairan He, Guanya Shi

CoRL 2026

TL;DR: SoFTA separates fast upper-body stabilization from slower locomotion control, helping humanoids keep their hands steady while walking, carrying nearly full cups, or recording video.

Whole-Body Model-Predictive Control of Legged Robots with MuJoCo overview

Whole-Body Model-Predictive Control of Legged Robots with MuJoCo

John Z. Zhang, Taylor A. Howell, Zeji Yi, Chaoyi Pan, Guanya Shi, Guannan Qu, Tom Erez, Yuval Tassa, Zachary Manchester

ICRA 2026

TL;DR: A simple combination of iLQR, MuJoCo dynamics, and finite-difference derivatives enables real-time whole-body control of physical quadrupeds and humanoids with few sim-to-real adjustments.

ASAP: Aligning Simulation and Real-World Physics for Learning Agile Humanoid Whole-Body Skills overview

ASAP: Aligning Simulation and Real-World Physics for Learning Agile Humanoid Whole-Body Skills

Tairan He, Jiawei Gao, Wenli Xiao, Yuanhang Zhang, Zi Wang, Jiashun Wang, Zhengyi Luo, Guanqi He, Nikhil Sobanbabu, Chaoyi Pan, Zeji Yi, Guannan Qu, Kris Kitani, Jessica Hodgins, Linxi "Jim" Fan, Yuke Zhu, Changliu Liu, Guanya Shi

RSS 2025

TL;DR: ASAP learns a residual action model from real-world data to correct simulation mismatch, then fine-tunes motion-tracking policies to enable more agile and coordinated humanoid skills.

In-Hand Manipulation teaser

In-Hand Manipulation of Unknown Shapes with Tactile Sensing

Chaoyi Pan*, Marion Lepert*, Shenli Yuan, Rika Antonova, Jeannette Bohg

IROS 2023

TL;DR: Tactile sensing builds an estimate of unknown object shape and pose, while Bayesian optimization balances exploration with reorientation for insertion, reducing exploration time in simulation.

Bimanual Handover teaser

Efficient Bimanual Handover and Rearrangement via Symmetry-Aware Actor-Critic Learning

Yunfei Li*, Chaoyi Pan*, Huazhe Xu, Xiaolong Wang, Yi Wu

ICRA 2023

TL;DR: Symmetry-aware actor-critic learning and object-centric goal relabeling help two arms coordinate multi-object handovers and rearrangement, with demonstrations on real robots and in human-robot collaboration.

Formation Control teaser

Flexible Decentralized Displacement-Based Formation Control: A Multi-Agent Reinforcement Learning Approach

Chaoyi Pan, Yuzi Yan, Zexu Zhang, Yuan Shen

EUSIPCO 2022

TL;DR: Multi-agent reinforcement learning with a Hausdorff-distance reward trains a decentralized policy for flexible robot formations without a shared global coordinate system, validated in simulation and on mobile robots.

Blog

View All Posts →

Projects

Agile Drone Transportation

Agile Drone Transportation with Explicit Time Trajectory Optimization

Trajectory optimization for quadrotor drones carrying slung payloads, enabling aggressive aerial transportation maneuvers through explicit time optimization and collision-aware planning.

Blind Navigation Headgear

Intelligent Blind Navigation Headgear Based on Head Conditioned Reflexes

A wearable headgear system that guides visually impaired users through head-conditioned reflex signals for intuitive, hands-free obstacle avoidance navigation.

Experience

Jan 2026 – Present

Amazon Frontier AI and Robotics Lab

Research Intern · San Francisco, USA

Working on robot data curation and annotation for pretraining. Supervised by Guanya Shi and Rocky Duan.

Jun – Aug 2025

Meta FAIR Embodiment & Actions Research Team

Research Science Intern · Pittsburgh, USA

Developed universal human-to-robot physics-based retargeting system for converting raw human interaction trajectories to feasible robot trajectories. Supervised by Francois Hogan.

Jun 2023 – Present

Carnegie Mellon University

PhD Student · Pittsburgh, USA

Developing generative models for control, bridging learning-based generative models with model-based control for contact-rich real-world tasks. Supervised by Prof. Guanya Shi and Prof. Guannan Qu.

Jun – Sep 2022

Stanford Interactive Perception and Robot Learning Lab

Summer Intern · Stanford, USA

Implemented tactile-based in-hand manipulation system using Bayesian optimization without object shape prior or vision information. Supervised by Prof. Jeannette Bohg.

Sep 2021 – Jun 2022

Tsinghua WuLab

Research Assistant · Beijing, China

Developed bimanual coordination system for handover and rearrangement tasks using structured reinforcement learning. Supervised by Prof. Yi Wu.

Oct 2020 – Sep 2021

Tsinghua SLab

Research Assistant · Beijing, China

Designed decentralized formation control system for mobile robots using multi-agent RL and Hausdorff distance. Supervised by Prof. Yuan Shen.