Thermal perception + language action + runtime safety

Safe-Night VLA: Seeing the Unseen via Thermal-Perceptive Vision-Language-Action Models for Safety-Critical Manipulation

Safe-Night VLA equips a pre-trained VLA policy with synchronized RGB, LWIR thermal, and depth observations, then constrains execution through a CBF-QP safety filter for thermodynamic and low-light manipulation.

Dian Yu* Qingchuan Zhou* Bingkun Huang Majid Khadiv Zewen Yang†

Munich Institute of Robotics and Machine Intelligence, Technical University of Munich

*Equal contribution. †Corresponding author: zewen.yang@tum.de.

This paper has been accepted to the 2026 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS 2026).

arXiv Code Full Demo BibTeX

Abstract

Robots should reason about what RGB cannot see.

Current VLA policies rely heavily on RGB observations, which makes thermodynamic state, buried targets, and optical deception hard to resolve. Safe-Night VLA adds LWIR thermal perception to a frozen vision-language backbone and trains only the action-side components, allowing the policy to ground prompts such as hot, cold, and buried in non-visible physical evidence.

The framework is paired with a runtime control barrier function filter. The VLA still proposes task-driven Cartesian intent, while the CBF-QP layer converts it into safe joint displacements under modeled workspace constraints. The paper evaluates this combination on a Franka robot across temperature-conditioned manipulation, subsurface localization, and reflection disambiguation.

Thermal, RGB, and depth observations for the three Safe-Night VLA scenarios
Thermal, RGB, and depth observations reveal complementary evidence across all three scenarios.

Method

A thermal-aware VLA stack with a hard safety layer.

01

Three synchronized views

RGB carries scene context, thermal pseudo-color exposes LWIR heat signatures, and depth supplies illumination-invariant geometry.

02

Frozen VLM backbone

The visual-language encoder is kept frozen to preserve pre-trained semantic structure while adapting to pseudo-color thermal cues.

03

Trainable DiT action head

The action head learns to translate multimodal tokens and language prompts into 6-DoF end-effector deltas plus gripper commands.

04

CBF-QP safety filter

A post-hoc quadratic program enforces joint limits and workspace barriers before robot execution.

Safe-Night VLA system architecture with multimodal input, frozen VLM, trainable DiT, and safety filter
System architecture: multimodal RGB-T-D input, frozen VLM, trainable diffusion transformer action head, and runtime IK plus CBF safety filter.

Interactive Framework

System architecture: multimodal RGB-T-D input, frozen VLM, trainable diffusion transformer action head, and runtime IK plus CBF safety filter.

Experiments

Three RGB failure modes, one physical benchmark.

Temperature-conditioned manipulation setup, training images, and runtime scene

Scenario I

Temperature-conditioned manipulation

The robot must pick the hot or cold bottle when the objects are visually similar, including runtime scenes with an unseen heated distractor.

Subsurface localization setup, training images, and runtime scene

Scenario II

Subsurface localization

A thermal bloom through granular media guides the policy toward a buried hot object that is almost invisible in RGB.

Illusion-rejection setup, training images, and runtime scene

Scenario III

Illusion rejection

LWIR attenuation in common glass helps reject mirror reflections that appear plausible in visible light.

Results

Thermal cues solve hidden semantics; safety filtering keeps execution bounded.

Dim/Night + Safety

64%

Hot/cold bottle success for Safe-Night VLA, compared with 0% for RGB-only.

Dim/Night + Safety

72%

Buried object localization success for the full RGB-T-D model.

Mirror Rejection

17/20

Safe-Night VLA successes under dim/night light with the safety filter enabled.

Attention Mass

53.5%

Hot-object attention mass with thermal input, versus 16.8% when thermal is masked.

Joint ablation of input modalities and runtime safety filtering under normal and dim/night illumination.
Task & Metric Normal Light Dim/Night Light
w/o Safety Filter w/ Safety Filter w/o Safety Filter w/ Safety Filter
RGB-OnlyRGB-DRGB-TOurs RGB-OnlyRGB-DRGB-TOurs RGB-OnlyRGB-DRGB-TOurs RGB-OnlyRGB-DRGB-TOurs
Success Rate (SR) ↑ (N = 50 trials)
1. Hot/Cold Bottle 32%24%78%72% 42%38%86%82% 0%10%10%56% 0%12%22%64%
2. Buried Object 14%24%60%74% 16%24%66%78% 0%0%42%68% 0%2%48%72%
Success Counts ↑ (N = 20 trials per sub-case)
3a. Single Box Success 13/208/2010/209/20 19/2017/2020/2017/20 0/200/204/2012/20 9/209/2017/2018/20
3b. Mirror Rejection Success 2/203/2012/204/20 12/2011/2019/2015/20 3/200/207/205/20 13/2011/2015/2017/20

Full Demo

Safe-Night VLA in action.

Click play to watch the full Safe-Night VLA robot demonstration.

Citation

BibTeX

@misc{yu2026nightvla,
      title={Safe-Night VLA: Seeing the Unseen via Thermal-Perceptive Vision-Language-Action Models for Safety-Critical Manipulation}, 
      author={Dian Yu and Qingchuan Zhou and Bingkun Huang and Majid Khadiv and Zewen Yang},
      year={2026},
      eprint={2603.05754},
      archivePrefix={arXiv},
      primaryClass={cs.RO},
      url={https://arxiv.org/abs/2603.05754}, 
}

License

MIT License

The website source code is released under the MIT License.

Unless otherwise stated, the paper, figures, videos, data, model weights, and other media assets are copyright of the authors and are not covered by the MIT License.