Voice Control for Robot Arms: Whisper in Front of a Policy

The demo is always the same. Somebody leans towards a laptop, says "pick up the red block", and a SO-100 arm reaches out and picks up the red block. It looks like the arm understood English. It did not. What happened is that a speech recogniser turned about two seconds of audio into a string, and that string was handed to a policy through exactly the same input slot that a command-line flag fills. If you had typed the sentence instead, the arm would have done the identical thing.

That distinction is the whole subject. Speech recognition is a text producer sitting in front of a language-conditioned policy. It is cheap, it is well understood, and it works. What it cannot do is give the policy a vocabulary it never learned, or make it care about words it was trained to ignore. This article covers both halves honestly: how to build the voice front end with software you can install today, and what remains broken on the other side of it.

What the policy actually receives when you talk to it

Audio never reaches the policy. Every vision-language-action model on this platform takes an image stack, a proprioceptive state vector and a text instruction. The instruction is a short plain-language sentence, and the model has its own opinion about how much attention to pay to it. The five trainable policies differ enormously here, and the difference decides whether voice control is even meaningful for your setup.

The per-step numbers come from the policy comparison. They matter later, when we work out where the recogniser can and cannot sit. For now the important column is the second one. If you trained ACT, there is no language input to speak into. You can build a beautiful voice pipeline and the arm will do the one thing it knows, every time, regardless of what you said.

# GR00T N1.7 policy input, from getting_started/policy.md in NVIDIA/Isaac-GR00T
observation = {
    "video": {...},              # camera frames
    "state": {...},              # joint positions
    "language": {
        "task": [["pick up the cube"]],   # shape (B, 1), list of lists of strings
    },
}

action, info = policy.get_action(observation)
# action has shape (B, T, D); the base checkpoint predicts an action horizon of 40 steps

# LeRobot deploys the same string through a flag
lerobot-rollout \
  --strategy.type=base \
  --policy.path=${HF_USER}/my_policy \
  --robot.type=so100_follower \
  --robot.port=/dev/ttyACM1 \
  --robot.cameras="{ up: {type: opencv, index_or_path: /dev/video10, width: 640, height: 480, fps: 30}}" \
  --task="Put lego brick into the transparent box" \
  --duration=60

The one-task dataset problem, which is where most voice projects die

Here is the failure that costs people a weekend. They record a LeRobot dataset the standard way, fine-tune SmolVLA, wire up Whisper, and discover the arm performs the same motion whatever they say. The transcript is perfect. The policy is trained. Nothing is broken. The dataset simply never contained more than one instruction.

Look at what the recorder actually writes. The dataset format stores a per-frame task, but the recording script fills it from a single flag that does not change for the duration of the session. What makes those episodes worth training on at all is a separate subject, covered in collecting high-quality VLA training data. Here we care about one field only: the one that carries the words.

lerobot-record \
    --robot.type=so101_follower \
    --robot.port=/dev/tty.usbmodem585A0076841 \
    --robot.id=my_awesome_follower_arm \
    --robot.cameras="{ front: {type: opencv, index_or_path: 0, width: 1920, height: 1080, fps: 30}}" \
    --teleop.type=so101_leader \
    --teleop.port=/dev/tty.usbmodem58760431551 \
    --teleop.id=my_awesome_leader_arm \
    --dataset.repo_id=${HF_USER}/record-test \
    --dataset.num_episodes=5 \
    --dataset.single_task="Grab the black cube"

# src/lerobot/scripts/lerobot_record.py
frame = {**observation_frame, **action_frame, "task": single_task}
dataset.add_frame(frame)

The result is a set of episodes whose task table has exactly one row. GR00T's data preparation guide shows what a multi-instruction table looks like by contrast, and the shape makes the point on its own.

// demo_data/cube_to_bowl_5/meta/tasks.jsonl in NVIDIA/Isaac-GR00T
{"task_index": 0, "task": "cube into yellow bowl"}
{"task_index": 1, "task": "cube into green bowl"}

The voice front end, component by component

Assume you have done the dataset work and your policy genuinely responds to different instructions. The audio side is now a well-trodden four-stage pipeline: gate on speech, optionally gate on a wake word, transcribe, then map the transcript onto a string the policy has actually seen. Every piece below is open source and was checked against its repository for this article.

1. 1
   Capture 16 kHz mono audio

   Everything downstream expects 16 kHz. Whisper resamples internally, Silero VAD supports 8000 Hz and 16000 Hz only, and openWakeWord wants frames in multiples of 80 ms, which is 1280 samples at 16 kHz. Pick 16 kHz once and stop converting.

   bash

   pip install sounddevice numpy
   # 1280 samples at 16 kHz = 80 ms, the openWakeWord frame size

2. 2
   Gate on voice activity so the recogniser never sees silence

   Silero VAD is about two megabytes as a JIT model and processes a 30 ms chunk in under 1 ms on a single CPU thread, under an MIT licence. This step is not an optimisation, it is a correctness fix: Whisper invents text when handed silence.

   python

   from silero_vad import load_silero_vad, read_audio, get_speech_timestamps
   
   model = load_silero_vad()
   wav = read_audio('utterance.wav')
   speech_timestamps = get_speech_timestamps(wav, model, return_seconds=True)

3. 3
   Add a wake word if the robot shares a room with humans

   openWakeWord suggests a false-accept rate below 0.5 per hour with a false-reject rate below 5 percent as reasonable in practice, and reports that a single core of a Raspberry Pi 3 can run 15 to 20 of its models simultaneously in real time. That budget matters when the same machine is also driving servos.

   python

   import openwakeword
   from openwakeword.model import Model
   
   openwakeword.utils.download_models()
   model = Model(wakeword_models=["path/to/model.tflite"])
   prediction = model.predict(frame)   # frame = 80 ms of 16 kHz audio

4. 4
   Transcribe the gated segment

   faster-whisper is the pragmatic default on a machine with a GPU. Its own benchmark on large-v2 reports 1m03s at fp16 against 2m23s for the reference openai/whisper implementation, and 59s at int8 with VRAM down from 4708 MB to 2926 MB.

   python

   from faster_whisper import WhisperModel
   
   model = WhisperModel("large-v3", device="cuda", compute_type="float16")
   segments, info = model.transcribe(
       "utterance.wav",
       beam_size=5,
       vad_filter=True,
       vad_parameters=dict(min_silence_duration_ms=500),
   )
   text = " ".join(s.text for s in segments).strip()

5. 5
   Snap the transcript onto a task string the policy has seen

   This is the step people skip and it is the one that decides whether the system works. Do not pass the raw transcript to the policy. Match it against the exact strings in meta/tasks.jsonl and refuse anything that does not match closely enough. A policy fed an unseen phrasing is out of distribution, and out of distribution means it does something, just not what you asked.

   python

   import json, difflib
   
   TASKS = [json.loads(l)["task"]
            for l in open("meta/tasks.jsonl") if l.strip()]
   
   def snap(transcript, cutoff=0.6):
       hit = difflib.get_close_matches(
           transcript.lower().strip(".!? "),
           [t.lower() for t in TASKS], n=1, cutoff=cutoff)
       if not hit:
           return None          # say "I do not know that command" and stop
       return TASKS[[t.lower() for t in TASKS].index(hit[0])]

6. 6
   Hand the canonical string to the running policy

   For LeRobot policies that is the --task value on the rollout, or the task field of the observation if you drive the policy from your own loop. For GR00T it is the language.task entry. Set it once at the start of an episode; do not rewrite it every control step.

   bash

   TASK=$(python voice_listen.py)   # prints one canonical task string, or exits non-zero
   
   lerobot-rollout \
     --strategy.type=base \
     --policy.path=${HF_USER}/my_policy \
     --robot.type=so100_follower \
     --robot.port=/dev/ttyACM1 \
     --task="$TASK" \
     --duration=60

Latency: a start-of-episode cost, not a control-loop cost

The most common design mistake is to treat the recogniser as part of the control loop. It is not, and it must not be. Inference latency on this platform runs from 20 ms per action step for ACT to 485 ms for Pi0.5. A streaming Whisper wrapper needs roughly 3.3 seconds to confirm a transcript. Those two numbers live in different worlds and should never share a loop.

Read the table the right way round. Three seconds of speech latency before the arm moves is fine, because a human said something and expects a pause. Three seconds inside an action chunk is a wrecked policy. So the architecture is: recognise, commit to a string, then start the rollout. If you want mid-episode interruption, implement it as a stop signal, not as a re-prompt. A stop can be a keyword spotter running at 80 ms granularity; a full re-transcription cannot keep up with a 20 ms loop and should not try.

Choosing a Whisper, with the numbers the repos publish

Whisper is OpenAI's open-weight speech recognition model, and the repository publishes a sizing table that is the only guide you need for picking one. The table below is that table, read from the upstream README in August 2026. For the wider story of how language got into robot policies at all, the vision-language-action overview is the background reading.

The turbo entry is worth understanding rather than just picking. The repository describes it as an optimised version of large-v3 that transcribes faster with a minimal degradation in accuracy, and the same README carries a blunt warning beside it: the turbo model is not trained for translation tasks, and it will return the original language even if you pass --task translate. For a fixed command vocabulary in one language that trade is almost free. If you ever need non-English speech turned into English, use one of the multilingual models instead.

# reference implementation
pip install -U openai-whisper
whisper audio.wav --model turbo

# GPU, roughly 2x faster than the reference at fp16 on the repo's own benchmark
pip install faster-whisper

# CPU or edge, C++ with quantisation
cmake -B build && cmake --build build -j --config Release
sh ./models/download-ggml-model.sh base.en
./build/bin/whisper-cli -m models/ggml-base.en.bin -f samples/jfk.wav

# quantise to shrink the model further
./build/bin/quantize models/ggml-base.en.bin models/ggml-base.en-q5_0.bin q5_0

Make the recogniser deaf to everything that is not a command

Whisper's defaults are tuned for transcribing podcasts, not for a microphone that is open all day next to a servo bus. The reference CLI defaults, read from the source, are worth knowing before you tune anything: temperature 0 with a fallback increment of 0.2, beam size 5, best of 5, condition_on_previous_text True, no_speech_threshold 0.6, logprob_threshold -1.0 and compression_ratio_threshold 2.4. Two of those want changing for robot use.

whisper utterance.wav \
  --model turbo \
  --language English \
  --condition_on_previous_text False \
  --no_speech_threshold 0.8 \
  --initial_prompt "pick up the red block. put the block in the bowl. open the gripper."

What speech recognition cannot fix

Assume the transcript is perfect. Every remaining failure lives in the policy, and there are four of them worth naming, because each has a different fix and none of the fixes is audio.

The second row is the one that looks like a language failure and is not. It is the same mechanism behind a policy that only works in one setup: the model latched onto whatever in the image correlated with success, and the instruction was never the cheapest signal available to it. That row deserves a number. When Vision Overrides Language (Fang and colleagues, February 2026) introduced LIBERO-CF to test whether policies still follow the instruction when vision and language disagree. Their inference-time mitigation, Counterfactual Action Guidance, raises language following accuracy on under-observed tasks by 9.7 percent for Pi0.5 with a training-free strategy, and in real-world evaluation reduces counterfactual failures by 9.4 percent. Read those as evidence of how much headroom is being lost to visual shortcuts in the first place. The third row has a number too: Learning What to Say to Your VLA (Jeong, Swamy and Bajcsy, June 2026) reports that searching over how the instruction is worded improves base VLA performance by 24.7 percent in simulation and 65.0 percent in hardware, on frozen pretrained policies with the weights untouched.

The contrast that makes the training side concrete is NVIDIA's own. On the same real GR-1 task, GR00T N1.5 follows the language command 93.3 percent of the time against 46.6 percent for N1, and on the Language Table benchmark 93.2 percent against 52.8 percent. That gap was closed by freezing and improving the vision-language model and changing the adapter, not by improving anybody's microphone. If your policy ignores words, the fix is upstream of the audio pipeline every single time. Comparing the language-capable models is a better use of an afternoon than tuning Whisper.

This distinction is worth being pedantic about, because the two things look identical in a video. Natural language over the platform's operations is a solved problem: an agent reads a tool schema and calls it, the same operations exposed by the CLI. Natural language over the arm's motion is not solved. It is a property of your training data, and it degrades exactly as far as your data is unbalanced. A high-level planner sits between the two: it decomposes a spoken instruction into a sequence of skills and calls them in order, but every skill in that sequence still has to have been learned from demonstrations first.

Recording a dataset that language can steer

If you want voice control to be more than a demo, the work is in the recording session, not the audio stack. The goal is a task table with several rows and enough demonstrations behind each row that the model has to read the words to predict the actions. The SmolVLA guide's own reference dataset is 50 episodes across 5 cube positions, 10 per position, and notes that 25 episodes was not enough. Treat that as a floor per instruction, not per dataset.

1. 1
   Design a command set where vision alone cannot disambiguate

   Two bowls of different colours in every frame, and the instruction picks one. If the yellow bowl is only ever present when the yellow instruction is given, you have rebuilt the shortcut and the model will use it.

2. 2
   Record one session per instruction

   Run lerobot-record once per task string. The flag is single_task and it applies to the whole session, so distinct instructions mean distinct invocations.

   bash

   lerobot-record ... --dataset.repo_id=${HF_USER}/two-bowls \
     --dataset.num_episodes=25 --dataset.single_task="cube into yellow bowl"
   
   lerobot-record ... --dataset.repo_id=${HF_USER}/two-bowls --resume=true \
     --dataset.root=/path/to/local/two-bowls \
     --dataset.num_episodes=25 --dataset.single_task="cube into green bowl"

3. 3
   Verify the task table before you spend money on a GPU

   Count the rows. This is a five second check that saves a four to twelve dollar training run and several hours.

   bash

   wc -l meta/tasks.jsonl
   cat meta/tasks.jsonl
   # expect one line per distinct instruction, not one line total

4. 4
   Convert the format if you are training GR00T

   The GR00T data preparation guide states the reason outright: GR00T currently uses the LeRobot v2 data format because many upstream datasets (DROID, LIBERO, Bridge) are published in v2. A v3.0 dataset has to be converted down first, with the conversion script that ships in the repo, and a rejected dataset is one of the more common training-time surprises.

   bash

   # LeRobot v3.0 -> v2, the script shipped in NVIDIA/Isaac-GR00T
   python scripts/lerobot_conversion/convert_v3_to_v2.py --help
   
   # then add meta/modality.json. The language block, from examples/SO100/modality.json:
   #   "annotation": {
   #       "human.task_description": { "original_key": "task_index" }
   #   }
   # the parquet column annotation.human.task_description is an int index into meta/tasks.jsonl

5. 5
   Fine-tune a language-capable policy

   SmolVLA on a 24 GB card is the cheapest way to test whether language conditioning took. The training docs describe the form and what the backend actually sends. Its minimum on this platform is 30 episodes and a run costs about 1 to 3 dollars. GR00T N1.7 and Pi0.5 need an A100 or H100 80 GB and land at about 4 to 12 dollars. The command below is the LeRobot doc's own example; the AY-Robots training form sends its own defaults instead.

   bash

   lerobot-train \
     --policy.path=lerobot/smolvla_base \
     --dataset.repo_id=${HF_USER}/two-bowls \
     --batch_size=64 \
     --steps=20000 \
     --output_dir=outputs/train/my_smolvla \
     --policy.device=cuda

6. 6
   Test with text before you test with a microphone

   Run the rollout twice with the two task strings and nothing else changed. If the arm does the same thing both times, stop. The audio pipeline cannot help and you will waste a day proving it.

   bash

   lerobot-rollout --strategy.type=base --policy.path=${HF_USER}/my_policy \
     --robot.type=so100_follower --robot.port=/dev/ttyACM1 \
     --task="cube into yellow bowl" --duration=60
   
   lerobot-rollout --strategy.type=base --policy.path=${HF_USER}/my_policy \
     --robot.type=so100_follower --robot.port=/dev/ttyACM1 \
     --task="cube into green bowl" --duration=60

Two routes to a talking arm

pip install faster-whisper silero-vad openwakeword sounddevice
pip install -e ".[smolvla]"       # in the lerobot checkout

# gate, transcribe, snap, launch
python voice_listen.py | xargs -I{} \
  lerobot-rollout --strategy.type=base \
    --policy.path=${HF_USER}/my_policy \
    --robot.type=so100_follower --robot.port=/dev/ttyACM1 \
    --task="{}" --duration=60

The short version

Speech recognition in front of a robot policy is a solved engineering problem with well-documented components and published numbers. Whisper and its faster reimplementations, a 2 MB voice activity detector, and an 80 ms wake word model will get you from a microphone to a canonical task string reliably, on hardware you already own, for the cost of an afternoon.

The unsolved part is on the other side of the string, and it is not an audio problem. It is a data problem, measurable before you buy a microphone by counting the rows in a JSON Lines file. A policy trained on one instruction cannot be steered by words. A policy trained on unbalanced scenes will follow the picture instead of the sentence. And ACT has no text input at all, which is worth knowing before you pick it because it is otherwise the cheapest and fastest of the five. Fix the dataset, then wire up the microphone. In that order the whole thing takes a weekend. In the other order it takes a month and does not work.

Frequently asked questions