Deformable Objects: Cloth and Cables on a Low-Cost Arm

A towel does not have a pose

Pick-and-place code carries an assumption nobody writes down: the thing you are grabbing has a state that fits in six numbers. Grasp planning inherits it, motion planning inherits it, and a learned policy that never sees those numbers still inherits it from the demonstrations, because rigid objects really do move as one piece when you pull them.

Fabric has none of that. The 2023 deformable-manipulation survey by Gu and colleagues opens on the reason: infinite dimensionality and complex dynamics compared with the rigid case. The practical consequence is not harder maths. It is that four parts of your pipeline stop being valid at the same time.

That last row is the expensive one. A bad grasp on a cube costs two seconds; a bad grasp on cloth leaves the fabric in a state nobody demonstrated, so the policy acts from an observation outside its training data. That is why cloth failures look like hesitation rather than collision, and why a policy freezing mid-motion is the usual symptom.

What the published cloth results actually say

The literature is more encouraging than most people expect and more discouraging than the demo videos suggest. The table lists only results I opened and read, with the hardware column taken from the same source as the number.

Now read the hardware column again. Not one platform has a single arm, and that is mechanical rather than accidental: a fold needs one hand to pin the fabric while the other moves it. SpeedFolding does not even infer the fold lines, they are supplied as user-defined instructions, and it still needs two arms to execute them.

Folding laundry and collecting laundry are different tasks

The Pi0.5 household demos get quoted as laundry results, and they are not quite that. In the Pi0 lineage, the Pi0.5 appendix defines Laundry Basket as plus one for navigating to and picking up the clothing, plus one for placing it into or on the basket, plus one for the clothing ending up fully inside. Nothing there requires knowing the shirt's shape. It is pick-and-place with a floppy object, and it is genuinely useful.

Folding is the other task. The pi0.6 model card of 17 November 2025 states that laundry folding and box assembly previously required fine-tuning with high-quality data to reach non-zero success rates. Take that as calibration: in late 2025, a 3-billion-parameter vision-language-action model on a bimanual research platform scored zero at out-of-the-box laundry folding.

The data number nobody quotes

The most useful cloth result of the past year is not a success rate, it is a failed ablation. LeRobot published both of its shirt-folding datasets to the Hub, so the numbers are checkable rather than quoted.

// lerobot/full_folding  meta/info.json
{ "codebase_version": "v3.0", "robot_type": "openarms_follower",
  "total_episodes": 5688, "total_frames": 14129038,
  "total_tasks": 1, "fps": 30 }

// lerobot/high_quality_folding  meta/info.json
{ "codebase_version": "v3.0", "robot_type": "openarms_follower",
  "total_episodes": 1200, "total_frames": 3254196,
  "total_tasks": 1, "fps": 30 }

Training on all 5,688 episodes gave 40% total success and 0% on the messy start. Fine-tuning the best checkpoint on the hand-picked 1,200 gave 90%: 100% flat, 80% messy. The curation rules were mundane, which is the point. Drop episodes with a poor final state, drop suspiciously short outliers, keep only episodes where the operator used one consistent technique.

X-VLA got to the same place from the other direction: retrain an ACT model after every 100 collected episodes, find its failure modes, record targeted demonstrations against them. Their appendix also gives the rate. One folding episode takes about 1.5 minutes, and an hour yields 20 to 25 episodes once resets and discarded attempts are counted, so 1,200 episodes is roughly 50 to 60 hours at the bench.

Three things that break on one low-cost arm

1. You only have one hand

The workaround is to make the table the second hand: pin one edge against a raised lip, drag with the jaws. That is why every realistic single-arm task below involves dragging, dropping, or folding against a fixed feature, and why fabric choice is load-bearing. A terry face towel on a rubber mat behaves almost like a semi-rigid object; a thin cotton t-shirt on a smooth desk behaves like a fluid.

2. The gripper cannot count layers

Grabbing exactly one layer is a research problem in its own right. Tirumala and colleagues put a ReSkin tactile sensor on one fingertip of a Franka arm and trained a classifier purely to decide how many layers were in the grasp, evaluated over 180 physical trials. The SO-100 has no tactile sensing and its end effector is a printed parallel jaw: two layers and one layer look identical in the joint stream. The LeRobot team redesigned their grippers with a larger contact surface and extended the upper arm by 5 cm before reaching 90%.

3. The state you need is not in the joint angles

An SO-100 recording carries an observation.state of six values (shoulder pan, shoulder lift, elbow flex, wrist flex, wrist roll, gripper) plus camera streams. On a rigid task a policy can partly cheat by replaying proprioceptive trajectories. On cloth there is nothing to cheat with: every fact about the object lives in the pixels, and the wrist camera becomes the sensor that decides whether the jaws closed on an edge or on the middle of the fabric.

What a single SO-100 can realistically do

This is a judgement call built from the results above, not a measured benchmark. Treat the middle column as guidance on where to spend your first 50 episodes.

Recording a deformable dataset that trains

The commands are the ordinary LeRobot dataset commands, checked against the current LeRobot imitation-learning docs. What changes for cloth is the protocol around them, not the flags.

1. 1
   Put a camera where the shape is

   One overhead camera that sees the whole fabric footprint, one wrist camera that sees what the jaws are about to close on. The LeRobot docs give the acceptance test: you should be able to do the task yourself looking only at the images. Run that test on a crumpled start, not a flat one.

   bash

   lerobot-teleoperate \
     --robot.type=so100_follower \
     --robot.port=/dev/ttyACM0 \
     --robot.id=my_so100 \
     --robot.cameras="{ top: {type: opencv, index_or_path: 0, width: 640, height: 480, fps: 30}, wrist: {type: opencv, index_or_path: 1, width: 640, height: 480, fps: 30} }" \
     --teleop.type=so100_leader \
     --teleop.port=/dev/ttyACM1 \
     --teleop.id=my_leader \
     --display_data=true

2. 2
   Write the rubric before you record anything

   Copy the pi0 approach and split the task into scored stages: reached, grasped an edge, lifted clear, released over the bin. Four spreadsheet columns, used at every evaluation from now on. It is the only way to see partial progress.

3. 3
   Record in blocks, with a reset budget that matches the fabric

   Record 50 episodes, not 400. Defaults are 60 s per episode, 60 s of reset and 50 episodes, already 100 minutes of clock time before a single retake. Use the reset window to re-crumple the fabric into a genuinely different configuration.

   bash

   lerobot-record \
     --robot.type=so100_follower \
     --robot.port=/dev/ttyACM0 \
     --robot.id=my_so100 \
     --robot.cameras="{ top: {type: opencv, index_or_path: 0, width: 640, height: 480, fps: 30}, wrist: {type: opencv, index_or_path: 1, width: 640, height: 480, fps: 30} }" \
     --teleop.type=so100_leader \
     --teleop.port=/dev/ttyACM1 \
     --teleop.id=my_leader \
     --dataset.repo_id=$HF_USER/so100_towel_to_bin \
     --dataset.single_task="Pick up the towel and drop it in the bin" \
     --dataset.num_episodes=50 \
     --dataset.episode_time_s=45 \
     --dataset.reset_time_s=30 \
     --display_data=true

4. 4
   Throw away your bad episodes while you are still at the bench

   Left arrow (or r) cancels and re-records the episode you just botched. That is the cheapest possible version of the LeRobot curation step: free now, expensive later. Add episodes afterwards with --resume=true, remembering that num_episodes then means additional episodes.

   bash

   # add 30 more episodes to the same dataset
   lerobot-record \
     --robot.type=so100_follower \
     --robot.port=/dev/ttyACM0 \
     --robot.id=my_so100 \
     --dataset.repo_id=$HF_USER/so100_towel_to_bin \
     --dataset.root=/data/so100_towel_to_bin \
     --dataset.num_episodes=30 \
     --resume=true

5. 5
   Train something fast, roll it out, let the failures pick the next block

   Short loop first. This is the DAgger-style cycle X-VLA used to get a competitive folding policy out of only 1,200 episodes. Do not collect a second large block until a rollout has told you which stage is broken.

   bash

   lerobot-train \
     --dataset.repo_id=$HF_USER/so100_towel_to_bin \
     --policy.type=act \
     --output_dir=outputs/train/act_towel \
     --job_name=act_towel \
     --policy.device=cuda \
     --policy.repo_id=$HF_USER/act_towel

6. 6
   Overlap inference if the model is slow

   Every lerobot-rollout strategy accepts --inference.type=rtc, real-time chunking, which generates the next action chunk while the current one executes. The LeRobot folding write-up reports rollouts at least twice as fast as synchronous inference.

   bash

   lerobot-rollout \
     --strategy.type=episodic \
     --inference.type=rtc \
     --policy.path=$HF_USER/act_towel \
     --robot.type=so100_follower \
     --robot.port=/dev/ttyACM0 \
     --task="Pick up the towel and drop it in the bin" \
     --duration=120

Which policy to point at a deformable task

The five trainable policies differ in ways that matter more than usual here, because cloth tasks are long-horizon and vision-dominated. These numbers are the platform's.

For cloth, inference latency is the least of your problems. A folding motion is quasi-static, so Pi0.5 at 485 ms spends time rather than failing, and this is the one task family where remote inference over the public internet is defensible. That is also the limit of the excuse: it works for slow pick-and-place, not for reactive motion, and a dynamic fling of the FlingBot kind dies on a round trip.

The argument for ACT is iteration speed, not final quality. It trains in hours on a 24 GB card, which is what the DAgger cycle needs, but it has no base model at all: it exists only after training on your task, so it starts with zero visual knowledge about fabric. X-VLA trained ACT from scratch on Soft-Fold and reported it failed to match the pretrained model's throughput. Use ACT to find failure modes, then move to GR00T N1.7 or Pi0.5 once the data is clean.

pip install lerobot

# record (full flags in the step list above)
lerobot-record --robot.type=so100_follower ... --dataset.num_episodes=50

# train something fast first
lerobot-train \
  --dataset.repo_id=$HF_USER/so100_towel_to_bin \
  --policy.type=smolvla \
  --policy.device=cuda \
  --output_dir=outputs/train/smolvla_towel

# run it on the arm
lerobot-rollout --strategy.type=base --inference.type=rtc \
  --policy.path=$HF_USER/smolvla_towel \
  --robot.type=so100_follower --duration=120

Cables fail differently from cloth

A deformable linear object is lower-dimensional than fabric but has a property cloth does not: topology. Two cable configurations can look nearly identical in an image and differ only in whether a crossing goes over or under, which decides whether your next pull tightens a knot or removes it. No amount of pixel-level policy learning resolves that from one view.

• Untangling needs explicit structure. HULK combined learned keypoints with a geometric planner and reached 97.9% over 378 simulation trials but 61.7% on the physical bilateral system, at 8.48 actions per trial.
• Routing needs touch. The tactile cable-routing work put a GelSight sensor on the fingertip and drove motion primitives from it, reporting that the tactile version significantly outperformed the same approach without it.
• Goal images work better for cables than for cloth. Deformable Ravens puts cables, fabrics and bags into one goal-conditioned Transporter Network benchmark, and cables are the tractable end of it.
• Connectors are rigid. A USB or barrel end has a pose. Model that part as insertion and let the flexible tail be a disturbance rather than a state you track.

What it costs to find out

A deformable pilot is cheap in GPU terms and expensive in bench hours, the opposite of most robot-learning projects. The GPU side is fixed and small; the recording side scales with how disciplined you are.

Start on the data collection side rather than the model side. If you have not built an arm yet, the three ways to start page includes driving a real one with no signup, and the SO-100 complete guide covers the build. The imitation learning hub covers the training side.

Take the ablation rather than the headline: 5,688 episodes gave 40%, 1,200 hand-picked ones gave 90%. Deformable manipulation punishes volume and rewards consistency, and consistency is the one lever a single low-cost arm has as much of as a research lab. Start with a towel, write the rubric first, and run your first policy on the easiest deformable task you can define.