Jetson Orin for Local VLA Inference: Memory, TensorRT, Speed

There is one honest reason to put an NVIDIA Jetson Orin next to a robot arm: the network round trip disappears. Every other argument is weaker than it sounds. An Orin will not turn a 3 B parameter vision-language-action model into a real-time controller, and getting a policy running on it takes a day of build tooling. What it does is remove the public internet sitting between a cloud endpoint and your servos.

Four things decide whether that trade is worth it: module memory, which policies fit, what TensorRT speeds up, and the throughput NVIDIA publishes for its own model on its own board. External numbers below were fetched in August 2026. Platform numbers come from the policy catalog.

The Orin lineup, in the numbers that decide things

Jetson modules are sold on TOPS figures, which are near useless here. What matters for a policy is how much memory the module has and how fast it is: a VLA is a memory-bandwidth problem long before a compute problem. Here is the Orin family as NVIDIA lists it, TOPS included only so you can ignore it deliberately.

The Orin Nano Super needs a note because its numbers changed under it: in December 2024 NVIDIA repriced the developer kit to 249 USD and shipped a software update raising the 8 GB module to 67 sparse INT8 TOPS and lifting memory bandwidth 50 percent to 102 GB/s. Any benchmark quoting 68 GB/s for an Orin Nano 8GB predates that and measures a slower board than the one you can buy.

Which of the five policies actually fit

This is where most Jetson plans die. The five policies you can train on AY-Robots span 80 M to 3 B parameters, exactly the range between an Orin Nano and an AGX Orin 64GB. Published memory requirements disagree between projects, so the table gives a source per figure.

The Pi0 row disagrees with itself. The LeRobot async guide says Pi0 occupies 14 GB at inference and SmolVLA about 2 GB; the openpi repository lists it as more than 8 GB with an RTX 4090 as the example. Both are true of different implementations and dtypes. Plan against 14 GB: being wrong the other way costs you a board. That repo also says it has only been tested on Ubuntu 22.04 and mentions Jetson nowhere, which tells you what upstream support to expect for Pi0.5 on an edge module.

Measured throughput: GR00T N1.7 on a Jetson Orin

NVIDIA publishes a per-device latency breakdown on the GR00T-N1.7-3B model card, at 4 denoising steps with one camera. Read the Jetson Orin rows column by column, because the columns say where the time goes.

Look at the backbone column. It does not move. TensorRT in that configuration compiles the diffusion transformer action head and leaves the vision-language backbone in PyTorch, so it buys what torch.compile already bought and nothing more. The Isaac-GR00T TensorRT page states it in those words: TensorRT optimizes only the DiT action head. The model card gives the reason it is the Orin setting specifically, which is that TensorRT 10.3 does not support the backbone engine on that board.

The deployment README in the same repo publishes a second table, from a newer full-pipeline export, and that one does move the backbone. Same board, same 4 denoising steps, same single camera. Which of the two you get on your own board depends on whether your TensorRT can build the backbone engine at all.

For scale: the AY-Robots catalog lists GR00T N1.7 at 152 ms per action step on its A100 or H100 tier, and a properly compiled Orin lands at 150.9 ms. That is not a claim that an Orin equals an A100; the figures come from different harnesses. It is a claim that at this model size the compute gap is far smaller than the price gap, and that the reason to go local is the network, not the silicon.

How TensorRT actually gets built on an Orin

The Isaac-GR00T Orin guide, as of August 2026, targets JetPack 7.2 with Jetson Linux 39.2, CUDA 13.2 and Python 3.12, on PyTorch 2.13.0 from the cu132 index. JetPack 6.x is no longer supported, so on an older image that flash is step zero.

1. 1
   Confirm the L4T baseline and unlock the clocks

   Check which Jetson Linux you are on, then move the module off its default power mode. Mode 0 is MAXN on AGX Orin, but on Orin Nano and Orin NX mode 0 is just a wattage default, so read the mode list before you pick one. Set the power mode first: after jetson_clocks runs, Jetson Linux requires a reboot before the mode can change again.

   bash

   cat /etc/nv_tegra_release
   sudo nvpmodel -q            # current mode; /etc/nvpmodel.conf lists them all
   sudo nvpmodel -m 0          # MAXN on AGX Orin; on Nano and NX pick the top mode id instead
   sudo jetson_clocks          # pin clocks to maximum, after the mode is set
   sudo tegrastats             # watch RAM, GPU load and thermals live

2. 2
   Install the Orin dependencies

   The repo ships a platform-specific installer. The Docker profile is recommended because it pins the CUDA and PyTorch combination for you.

   bash

   # bare metal
   bash scripts/deployment/orin/install_deps.sh
   
   # or the container
   cd docker && bash build.sh --profile=orin && cd ..

3. 3
   Activate the environment the aarch64 way

   The step people skip. On aarch64, do not use uv run: it re-syncs against the x86_64 root configuration and quietly replaces your locally built flash-attn wheel.

   bash

   source .venv/bin/activate
   source scripts/activate_orin.sh
   
   # then plain python, never `uv run python`
   python -c "import gr00t; print('ok')"

4. 4
   Build the TensorRT pipeline

   The build takes a checkpoint plus a sample dataset to trace real input shapes. The deployment guide puts the engine build at roughly 2 to 5 minutes depending on the GPU; the ONNX export in front of it is what makes the first run long. The export defaults to bf16 and bakes batch size 1 in statically.

   bash

   python scripts/deployment/build_trt_pipeline.py \
     --model-path checkpoints/GR00T-N1.7-LIBERO/libero_10 \
     --dataset-path demo_data/libero_demo \
     --embodiment-tag LIBERO_PANDA

5. 5
   Run and compare the two modes

   Run the same trajectories through PyTorch and through the engines. If TensorRT is not meaningfully faster than torch.compile, your build did not cover the backbone and you have the 216 ms number, not the 150 ms one. On a TensorRT that cannot build the backbone engine, --inference-mode tensorrt (DiT-only) is the mode that will load.

   bash

   python scripts/deployment/standalone_inference_script.py \
     --model-path checkpoints/GR00T-N1.7-LIBERO/libero_10 \
     --dataset-path demo_data/libero_demo \
     --embodiment-tag LIBERO_PANDA \
     --traj-ids 0 1 2 3 4 \
     --inference-mode trt_full_pipeline \
     --trt-engine-path ./gr00t_trt_deployment/engines

• Engines are GPU-specific and have to be rebuilt for a different GPU. The docs' own example is that an engine built on an RTX 4090 will not work on an H100, so the one you built while prototyping is not the one you ship.
• Batch size is baked into the export and cannot be changed at runtime.
• Orin and Thor share one flash-attn wheel carrying kernels for both sm_87 and sm_110, so a wheel that looks wrong for your board may be right.
• If the module boots from eMMC, point HF_HOME at an NVMe SSD before downloading a 3 B checkpoint.

The models that are genuinely comfortable on an Orin

The honest Orin shortlist is two models. ACT is roughly 80 M parameters, trains from scratch on your task, and the catalog puts it at 20 ms per action step on a 24 GB card. SmolVLA is 450 M parameters, about 100 M of that the action expert, on a SmolVLM-2 backbone. Its paper says it was designed to be trained on a single GPU and deployed on consumer-grade GPUs or even CPUs, and reports 40 percent faster training and 6x less memory than Pi0.

Third-party Orin measurement is thin. The NanoVLA paper benchmarks on a Jetson Orin Nano Super developer kit, the 8 GB 67 TOPS board, using SmolVLA at 50 action-chunk steps as its baseline and reporting 43.8 percent higher FPS than it, plus 52x higher FPS than OpenVLA on LIBERO-Goal. Take the baseline choice as the finding: SmolVLA is what people get running on the small board.

The latency arithmetic that actually governs the arm

One action per 150 ms forward pass gives a 6.6 Hz arm, which is visibly hesitant. What rescues it is action chunking: each pass emits a whole horizon of future actions and the robot plays them out at its own rate while the next chunk computes. The AY-Robots ACT trainer defaults to a chunk size of 100 with 100 action steps, so one inference covers over three seconds of 30 Hz motion.

The ceiling column is 1000 divided by the catalog latency, and those latencies came from the platform's cloud tiers, not an Orin. Use them as the shape of the problem, not as Jetson numbers. Either way, everything except ACT depends on chunking, and how you stitch chunks together separates smooth motion from a stutter every time one lands. That looks like a policy freezing mid-motion, so rule out the boring cause before blaming the model.

LeRobot ships two answers and they compose. Async inference splits a PolicyServer from a RobotClient over gRPC so the arm keeps executing while the next chunk computes, defaulting to actions_per_chunk=50 and a chunk_size_threshold the docs list as 0.7 while recommending 0.5 to 0.6. Real-Time Chunking, from Black, Galliker and Levine (arXiv 2506.07339), blends the start of each new chunk into the tail of the one already running. On an Orin you want both, and both can run on the same board.

# Run a trained policy on the arm with real-time chunking,
# with the policy on the Orin's own GPU.
lerobot-rollout \
  --strategy.type=base \
  --inference.type=rtc \
  --inference.rtc.execution_horizon=10 \
  --inference.rtc.max_guidance_weight=10.0 \
  --policy.path=/home/nvidia/checkpoints/smolvla_so100 \
  --robot.type=so100_follower \
  --robot.port=/dev/ttyACM0 \
  --robot.cameras="{ front: {type: opencv, index_or_path: 0, width: 640, height: 480, fps: 30}, wrist: {type: opencv, index_or_path: 1, width: 640, height: 480, fps: 30}}" \
  --task="Put the lego brick into the box" \
  --fps=30 \
  --duration=120 \
  --device=cuda

What makes this debuggable is that lerobot-rollout reports what the loop achieved rather than what you asked for: effective cadence, the share of cycles that blew the per-tick budget, a per-step breakdown, and pacing headroom. A high infer share points at the policy or the device; a high observe share points at your cameras, which on a Jetson is more common than people expect.

Cadence summary - whole run, 2 episodes - target 30 Hz (33.3 ms budget per tick)
  effective cadence: 29.84 Hz policy over 40.2 s measured
  cycles over the 33.3 ms work budget: 12/1197 (1.0%) - work mean 18.7 ms, worst 48.1 ms
  loop-body steps (share of measured work):
    observe      mean   3.13 ms - worst   9.00 ms -  31.7% of work
    infer        mean   5.18 ms - worst  40.00 ms -  52.4% of work
    send         mean   0.40 ms - worst   0.40 ms -   4.1% of work
  pacing headroom: 7.4 ms slept per tick on average - near zero means the loop is saturated

Two routes to the same running policy

One note for working from a terminal on the Orin: the operations exposed by the CLI are also exposed to AI agents through the MCP server. Neither runs inference on your board; both move datasets, runs and checkpoints around.

When an Orin is the wrong answer

The case against deserves stating, because the failure mode is spending the price of an AGX and three days of build tooling to move a policy from 5 Hz to 6.6 Hz.

• You are still iterating on the dataset. Every checkpoint change means a fresh engine build. Iterate against a cloud pod; move to the Orin when the policy stops changing.
• Your task is slow pick-and-place. A remote endpoint is genuinely viable there.
• You picked Pi0.5 or GR00T and your budget is an 8 GB board. Change the model, not the board.
• Your bottleneck is the cameras. If observe dominates the cadence report, a faster GPU changes nothing.
• You want to fine-tune on the same machine. No Orin has 40 GB, and out-of-memory during training is the most predictable failure here.

The counter-case: a frozen checkpoint on a SO-100 on a repetitive task where the round trip matters. That is the Orin's job. If you are not that far yet, run your first policy against a cloud pod, or drive a real arm at /live with no signup first. Background on the model families is in the VLA overview, and the hardware side in the SO-100 guide.