Beyond the End-Effector:
Learning Whole-Arm Contact Manipulation
from Human Demonstrations

The hand holds a plate level while the forearm pushes the cabinet door shut.

The hand keeps a cup upright while the forearm opens the coffee machine.

The hand keeps a wet pad above the sink while the arm turns the faucet off.

The hand stays tied to a fixed point while the arm presses the button.

A contact trajectory learned from bare-hand humans, paired with path constraints written from language, lets a robot manipulate with its whole arm while its hand stays committed to another task, with no teleoperation and no retraining.

Abstract

Robot manipulation has traditionally relied on end-effector-centric interactions. In contrast, humans routinely exploit whole-arm contacts to improve efficiency, robustness, and adaptability, especially when the hands are occupied or when precise grasping is unnecessary. Enabling similar capabilities in robots remains challenging: the robot must select where on its arm to make contact, generate effective non-prehensile interactions, and satisfy task-specific pose or safety constraints on the end-effector.

We propose Whole-Arm Contact Learning (WACL), a hierarchical framework that carries each whole-arm contact task to a trajectory optimizer through three robot-agnostic representations: semantic keypoints that summarize the scene, a reference contact trajectory that specifies where and how the arm presses on the surface, and safety-aware path constraints, generated from language by vision-language models, that protect the end-effector's task. The robot's kinematics enters only in a downstream surface-contact optimizer, which grounds these representations onto the arm by jointly searching over which link makes contact, where on that link, and the joint configuration that realizes it.

This decomposition lets WACL learn the contact trajectory from a handful of bare-hand human demonstrations, while a new end-effector constraint changes only the optimizer's input rather than any trained component. Real-world experiments on a 7-DoF arm across four whole-arm tasks show that WACL sustains contact under end-effector constraints where a strong keypoint-conditioned baseline fails (78% vs. 33% success), generalizes to new constraints without retraining, and learns without teleoperation.

Why Whole-Arm Contact?

In daily life, manipulation is rarely confined to the fingertips. When our hands are already occupied — carrying a load or holding a cup of coffee — we recruit the rest of the arm: nudging a door open with an elbow, steadying a box against a forearm.

This is a two-task problem: the hand keeps the cup upright (the primary task) while the arm presses a button or pushes a drawer shut (the secondary task). Whatever the arm does must not disturb what the hand is already committed to.

Robot manipulation, however, remains overwhelmingly end-effector-centric. Pre-trained VLAs output gripper-pose actions and treat the arm as a transport mechanism, so arm-environment contact is usually avoided rather than exploited. This discards useful kinematic redundancy and limits the robot in constrained settings where a gripper-only motion may be infeasible. Learning whole-arm manipulation end-to-end is difficult for two compounding reasons.

Coffee task
Cabinet task
Faucet task
Button task

Within-robot generalization gap

The same contact task looks entirely different depending on which link executes it — different pixels, different joint angles. A policy trained on raw observations of one link cannot transfer the contact behavior to another without retraining.

Human-to-robot embodiment gap

People show whole-arm contact naturally, but controlling a 7-DoF arm into the same positions via teleoperation demands a sophisticated leader-follower system and is slow and unintuitive. Bare-hand demonstrations are easier to collect but embodiment gaps remain.

Method

From Human Demonstrations to Contact Trajectories

Teleoperating a 7-DoF arm into whole-arm contact is slow and unintuitive. WACL instead learns from bare-hand demonstrations recorded in front of calibrated RGB-D cameras — no robot, no teleoperation hardware. From these it distills semantic keypoints and a reference contact trajectory: where on the object the arm should press and which way the surface faces. Because neither representation names a particular link, the learned behavior is robot-agnostic.

A person shows the task bare-handed,
with no robot and no teleoperation.

The robot learns to open the coffee machine with its forearm while the hand keeps the cup upright.

Whole-Arm Contact Learning

The contact trajectory is one of three robot-agnostic abstractions WACL passes to a surface-contact optimizer. The other two are semantic keypoints that summarize the scene and VLM-generated safety-aware path constraints that encode what the primary end-effector task demands — keep a cup upright, stay above the sink. The robot's kinematic chain enters only here: the optimizer jointly searches over which link makes contact, where on that link, and the joint configuration, so a new end-effector requirement changes only the optimizer's input, not any trained component.

WACL framework: three robot-agnostic representations grounded by a single surface-contact optimizer
1

Contact Trajectory Learning

From a handful of bare-hand human demonstrations, WACL leverages KALM to distill semantic keypoints and learn a reference contact trajectory: where on the object to make contact and the surface normal to follow. It commits to no particular link, so the behavior stays robot-agnostic.

2

Safety-Aware Path Constraints

A vision-language model turns the task and scene into safety-aware path constraints on the rest of the arm, such as keep a held cup upright or stay above the sink. They are regenerated from language on every call, so a new task needs no retraining.

3

Surface-Contact Optimization

Only here does the robot's kinematics enter. A surface-contact optimizer searches jointly over which link makes contact, surface anchors on that link, and the joint configuration, assembling a collision-free trajectory that tracks the contact target while every constraint holds.

Baseline Comparison

WACL and the baseline learn from the same backbone, the keypoint-conditioned diffusion policy KALM. The difference is what that policy commits to. The baseline locks in one full arm motion and replays it, so when a new rule on the hand makes that motion impossible, it has no fallback. WACL instead only specifies where to touch the object, and lets the final step choose which part of the arm touches and how. That freedom is what keeps it working when the hand is constrained.

Task success when the hand is constrained
78% WACL (ours)
vs
33% KALM baseline
Task Unconstrained Constrained EEF Task Constraints
KALM WACL (ours) KALM WACL (ours)
Open coffee machine 4/106/10 2/107/10 Hold the cup upright
Close cabinet door 7/107/10 5/108/10 Hold the plate horizontal
Close faucet 6/108/10 3/107/10 Stay above the sink
Press button 6/109/10 3/109/10 Stay within sphere
Aggregate 58%75% 33% 78%
✓ WACL (ours)
WACL keeps the cup upright the whole time, routing the contact through the forearm to open the coffee machine.
✗ KALM baseline
The baseline replays one fixed arm motion. It can barely succeed even without the cup constraint.

Generalization

Contact Regions

WACL jointly optimizes the contact link, the contact surface point, and the joint configuration to find the best way to complete the task while satisfying all constraints. In the three runs below it presses the same button from three different object positions and joint configurations. It uses different parts of the arm for each run.

New Constraints

WACL generates the end-effector constraints at inference, so changing the end-effector task is as simple as adding a new constraint, with no training required. Here we make the faucet task harder by swapping the pad for a cup: now WACL must keep the cup upright and hold it above the sink at the same time.