Tactile Sensing for Low-Cost Robot Arms: What Touch Adds

Touch is the one modality the low-cost stack skipped

An SO-100 ships with joint encoders and whatever USB cameras you bolt onto it. That is its entire sensory world. Contact is never measured, only inferred: the policy watches a gripper close on a pixel blob and guesses something is held. When the guess is wrong the arm keeps executing a grasp it does not have, and you get a gripper that does not close on the thing it was aimed at.

Tactile sensing is supposed to fix that, and the hardware has existed for a decade. The useful question is narrower than "is touch good": what can you buy today, how does the signal get into a LeRobot dataset, and can any of the five policies you can actually fine-tune read it.

What a hobby-price tactile sensor actually is

Two families dominate at this price. Vision-based sensors put a tiny camera behind a silicone pad lit by coloured LEDs and read contact as a picture of the deformed gel. Magnetic skins mix magnetic particles into an elastomer and read deformation as flux change at magnetometers underneath. One gives you an image with microns of detail; the other a short vector at hundreds of hertz.

The camera-in-a-fingertip family

GelSight is the original. The 2017 Sensors paper describes a fingertip sensor with an 18 by 14 mm sensing field and a spatial resolution "around 20 to 30 microns", a minimum perceivable force mostly under 0.05 N, and CNN force regression with R-squared above 0.9. Read that range carefully: the same paper puts compact GelSight devices for robot fingers at 30 to 100 microns generally, and 1 to 2 microns only for builds optimised for resolution over size. DIGIT shrank the format to 20 by 27 by 18 mm and 20 grams, running an Omnivision OVM7692 at 640 by 480 and 60 fps for about 15 USD of parts at a batch of 1000.

The catch is the gel. GelSight rates the Mini's 4.25 mm Lambertian silicone cartridge at 1000 coin presses, 0 to 30 degrees C, and 25 fps. That is slower than the 30 fps most people record an episode at, so the streams will not line up without resampling.

The magnetic skin family

ReSkin takes the opposite trade. Five MLX90393 magnetometers, four spaced 7 mm around a central one, cover 20 by 20 mm and stream 20 values (temperature plus Bx, By, Bz per chip) at roughly 400 Hz. The paper claims under 30 USD, a 2 to 3 mm skin, 1 mm contact localisation at 90 percent accuracy, and a comparison table rating the skin durable beyond 50,000 contacts. Read that last one as a claim that the learned model still predicts, not a wear-out point: the same paper runs 50,000 indentations and watches error climb unless the no-load baseline is re-measured. AnySkin rebuilds ReSkin as a self-adhering 2 mm skin that swaps in 12 seconds on average, and reports a policy losing only 13 percent when moved to a different physical skin.

None of the five trainable policies has a tactile input

Worth being blunt about. Across the five policies you can fine-tune here the observation space is identical: camera images, a joint-state vector, and for the VLA models a language instruction. No touch channel, no force channel, no contact flag. A vision-language-action model is called that for a reason.

The door is not shut. The signal just has to disguise itself as something the policy already accepts. Here is where each one lets you put it.

The two encodings people actually use

You do not have to take this on trust. Public SO-101 datasets with an AnySkin on the gripper, recorded at 30 fps in LeRobot v2.1, show both encodings.

Encoding 1: tactile as a fake camera

The magnetic readings are rendered into a picture and written as an ordinary video feature. To the loader it is a camera. To the policy it is a camera. Nothing in the training code knows otherwise.

// meta/info.json, oordonez/single_arm_so101_USB_anyskin
// robot_type so101_follower, fps 30, 10 episodes, 5126 frames, LeRobot v2.1

"observation.state":             { "dtype": "float32", "shape": [6] }
"action":                        { "dtype": "float32", "shape": [6] }
"observation.images.global_top": { "dtype": "video",   "shape": [720, 1280, 3] }
"observation.images.front":      { "dtype": "video",   "shape": [480, 640, 3] }
"observation.images.anyskin1":   { "dtype": "video",   "shape": [400, 400, 3] }

Encoding 2: tactile as extra state dimensions

The second route appends the raw magnetometer values to the proprioceptive vector. Five magnetometers times three axes is 15, and 6 joints plus 15 is 21. The same author published the matched ablation with those dimensions removed, same task and same 3365 frames.

// oordonez/rev3_ACT_single_arm_so101_USB_anyskin_raw  (10 episodes, 3365 frames)
"observation.state": { "dtype": "float32", "shape": [21] }   // 6 joints + 15 tactile

// oordonez/rev3_ACT_single_arm_so101_USB_NO_anyskin   (10 episodes, 3365 frames)
"observation.state": { "dtype": "float32", "shape": [6] }    // the control run

// trained config.json, both runs: "type": "act", chunk_size 100, n_action_steps 100

Ten episodes is far below what these policies need. The minimum here is 30 for SmolVLA and 50 for ACT, GR00T N1.7 and Pi0.5. Treat those datasets as a schema reference, not evidence that tactile helped.

Getting a tactile stream into a LeRobot recording

LeRobot has a documented plugin path and it treats tactile sensors as cameras. Any installed package named lerobot_camera_ is auto-discovered and the camera type it registers works from the CLI like a webcam. The only tactile plugin listed in the upstream docs today is one for Xense sensors, registering the type xense with rectified, difference, depth, marker, force and mesh outputs. Check its README before planning around that list: a LeRobot robot camera expects a single HWC image, so only the image-shaped outputs travel down the normal recording path and the force fields stay reachable only from your own code. For DIGIT, GelSight or AnySkin, you write the shim yourself.

1. 1
   Get the sensor talking on its own first

   Connect the AnySkin magnetometer board to the Adafruit QT Py over the QWIIC cable, find the port, run the visualiser. Press B to re-zero when the baseline drifts.

   bash

   pip install anyskin
   
   # Linux
   ls /dev/ | grep -e ACM -e USB      # usually /dev/ttyACM0
   sudo chmod a+rw /dev/ttyACM0
   
   # macOS
   ls /dev/ | grep cu.usb            # usually cu.usbmodem*
   
   anyskin_viz /dev/ttyACM0

2. 2
   For an optical sensor, confirm the raw stream

   DIGIT enumerates as a USB camera. The archived digit-interface package still installs and defaults to VGA at 30 fps. GelSight Mini has its own GPL-3.0 SDK with live-view and marker-tracking demos.

   python

   # pip install digit-interface
   from digit_interface import Digit
   
   d = Digit("D12345")   # serial number printed on the sensor
   d.connect()
   frame = d.get_frame()  # numpy array, default VGA 640x480 @ 30fps
   d.show_view()
   d.disconnect()

3. 3
   Decide the encoding before you record

   Image or state. If image, resize the tactile frame to match your cameras. If state, extend the robot's observation_features so the extra dimensions are named.

   python

   @property
   def observation_features(self) -> dict:
       return {**self._motors_ft, **self._cameras_ft,
               "tactile": (15,)}   # 5 magnetometers x 3 axes
   
   def get_observation(self) -> dict:
       obs = super().get_observation()
       obs["tactile"] = self.skin.get_sample()
       return obs

4. 4
   Record with the sensor named like a camera

   The documented plugin invocation. Note the Xense plugin is not on PyPI, so it installs from a clone, and its config takes a sensor serial number and output types rather than width and height. Swap in whatever your own plugin registers.

   bash

   # the generic pattern from the LeRobot docs
   pip install lerobot_camera_<name>
   
   # the Xense plugin specifically: not on PyPI, install the SDK then the clone
   pip install "xensesdk>=2.0.0"
   git clone https://github.com/xensedyl/lerobot-camera-xense
   pip install -e lerobot-camera-xense
   
   lerobot-record \
     --robot.type=so101_follower \
     --robot.port=/dev/ttyACM0 \
     --robot.cameras="{ front: {type: opencv, width: 640, height: 480, fps: 30},
                        tactile: {type: xense, serial_number: OG001319,
                                  output_types: [rectify], fps: 30} }" \
     --dataset.repo_id=${HF_USER}/so101-tactile \
     --dataset.num_episodes=50

5. 5
   Read the feature table before spending GPU money

   The cheapest bug-catch in the pipeline. Heading for GR00T, convert a v3.0 dataset to v2.1 now.

   bash

   python -c "import json;d=json.load(open('meta/info.json'));\
   [print(k, v['dtype'], v['shape']) for k,v in d['features'].items()]"

The honest limit: action chunking throws most of the signal away

Suppose you solved all of that and the fine-tuning run converges. There is still a structural problem, and it is not the sensor. It is action chunking. These policies do not run a closed loop at sensor rate. They look once, emit a chunk of future actions, and execute that chunk open loop.

Now compare the phenomenon you bought the sensor for. AnySkin streams at 100 Hz and ReSkin at roughly 400 Hz because slip is a tens-of-milliseconds event. At ACT's default nActionSteps of 100 the arm commits to about three and a third seconds of motion from one glance at the skin, and every tactile sample inside that window is recorded and ignored. Lowering nActionSteps below chunkSize buys more frequent re-planning at the cost of more inference calls. That is the knob that decides whether touch can close a loop at all.

Two ways to run the experiment

What the published work claims touch buys you

The research side is healthier than the tooling. The useful results cluster into four claims, and measured is worth separating from asserted.

• Slip detection works and is cheap. AnySkin reports 92 percent accuracy on unseen objects with an LSTM over 100 Hz magnetic data, the one result reproducible on an SO-100 in an afternoon.
• Tactile representations transfer. Sparsh pretrains on over 460,000 tactile images and reports 95.1 percent average improvement over task-specific end-to-end training on the six-task TacBench suite.
• Sensor instances are interchangeable, which used to be the blocker. AnySkin shows a 13 percent drop when the physical skin is swapped, in 12 seconds.
• Fusing touch into a VLA is a research direction, not a shipped feature. Tactile-VLA (July 2025) adds a hybrid position-force controller and a tactile reasoning module, arguing the vision-language backbone already holds semantic knowledge about physical interaction. A fork, not a dropdown.

That is the honest summary as of August 2026. Browse the Arena and the pattern holds: the models that got scaled, released and benchmarked are vision-language-action models. Tactile-augmented variants publish encouraging numbers; none is a checkpoint you can fine-tune this afternoon. Compare how fast VLAs themselves went from paper to downloadable weights.

The tactile signal you already own

Before spending anything, there is a crude force proxy already on the arm. The STS3215 exposes a read-only Present_Load register and lerobot's Feetech table has the address. It is not tactile sensing: it is whole-joint torque with the gripper's own friction folded in, and it says nothing about where contact happened. But it costs nothing and tells you whether the gripper is pressing on something.

# from lerobot/src/lerobot/motors/feetech/tables.py (STS/SMS series)
#   "Present_Load":    (60, 2)   # address 60, 2 bytes, read-only
#   "Present_Current": (69, 2)   # address 69, 2 bytes, read-only
#   sign-magnitude encoding: Present_Load uses bit 10 as the sign bit
#   sts3215 model number: 777

load = bus.sync_read("Present_Load")      # motor name -> signed value
grip = load["gripper"]
contact = abs(grip) > threshold           # calibrate the threshold per arm

What we would actually do

Start with Present_Load and an ablation. If that moves the needle, buy a 128 USD AnySkin fingertip rather than a 355 USD DIGIT: a 2 mm skin does not change gripper geometry or force a re-calibration of the arm. Record 50 episodes with and without, then train ACT on the SO-100 both ways. ACT is the right first probe: cheapest run here, trained from scratch so no pretrained prior fights your new input dimensions, and at 20 ms per action step the only one of the five fast enough for a reactive loop to mean anything.

Only then consider an optical sensor, and only if the task needs surface geometry rather than contact and slip. With a 100-step chunk you are giving the policy tactile context when it plans, not tactile feedback while it moves. If the arm is not built yet, the SO-100 setup guide and the recording tutorial come first, and /live lets you drive a real arm with no signup before you buy anything.