One Scan is Enough: Demonstration-Free Adaptation for Language-Guided Navigation

Overview

Method

Scan2Instruction: compiling a scan into language-navigation episodes

A handheld iPhone with a built-in LiDAR sensor captures the scene as an RGB-D video, with per-frame camera poses from ARKit or offline ORB-SLAM3. Scan2Instruction recovers two coupled layers from it: an appearance layer, a planar-regularized 3D Gaussian splat reconstructed with LiDAR depth supervision that renders photorealistic first-person views, and a geometric layer, a bird's-eye free-space map with a continuous clearance field. A* with a clearance-penalized edge cost plans collision-aware reference trajectories between sampled start and goal points, and a pretrained MLLM turns rendered route frames and goal views into three instruction types.

RealNav: a continuous-trajectory navigation VLA

RealNav is built on Qwen3-VL. The instruction and a sampled history of RGB observations pass through the backbone together with a small set of dedicated navigation tokens, whose final hidden states feed a lightweight MLP action head. The head predicts K = 8 future local waypoints (x, y, θ) in the robot frame. RealNav is first pretrained on R2R-CE and RxR-CE with an MSE behavior-cloning loss, before any access to the target scene.

PRPT-KS: Pareto-utility Reinforcement Post-Training in Known Scenes

Because the target scene is already known, PRPT-KS reads a closed-form three-axis Pareto utility U = (U_prog, U_safe, U_stop) of path progress, obstacle safety, and correct stopping directly off the reconstructed geometry. The same vector is used three ways:

• Pareto-utility advantage: the sum of the three components replaces a learned critic in a clipped policy-gradient update;
• Pareto-admissible rollouts: a per-axis test lets the planner replace actions that leave the free-space map or regress progress;
• Pareto-weighted preference pairs: each rejected action paired with the planner action becomes a preference label, weighted by vector Pareto dominance.

A step-wise expert buffer of planner reference actions supplies a behavior-cloning anchor, and a short-to-long trajectory curriculum and a cosine covariance schedule keep post-training stable. The unified objective combines the policy-gradient loss, the behavior-cloning anchor, and the Pareto-weighted preference loss.

Results

We evaluate on five real scenes (an office, a cottage, a teaching building, and two apartments; 768 m² scanned in 62 minutes in total), both in the reconstructed Gaussian scenes and on the physical robot. Human-demo-sft variants are fine-tuned on 20 teleoperated trajectories per scene.

Main evaluation on five scanned scenes

Success rate (SR) as sim / real. Average SR is computed from pooled successes and trials across scenes.

In 100 real-robot trials in never-scanned regions of the same buildings, where starts, goals, and waypoints all lie outside the reconstruction, RealNav + PRPT-KS reaches 0.82 SR, compared with 0.41 for RealNav (human-demo-sft) and 0.40 for CorrectNav (human-demo-sft).

Ablations

The full pipeline compared with RealNav before scene-specific post-training and with variants that each disable one loss term of the post-training objective. Metrics are measured on the reconstructed Gaussian scenes.

BibTeX

@inproceedings{yu2026onescan,
  title     = {One Scan is Enough: Demonstration-Free Adaptation for Language-Guided Navigation},
  author    = {Yu, Zhuoyuan and Long, Yuxing and Wang, Du and Wang, Junyang and Fan, Hongwei and Dong, Hao},
  booktitle = {Advances in Neural Information Processing Systems},
  year      = {2026}
}