publications | Jiaqi Peng

2026

X-NavDP: Generalizing Navigation Diffusion Policy to Novel Behavior and Embodiments with Group Q-score Reweighted Matching

Tianyu Yang, Yiming Zeng, Wenzhe Cai, and 5 more authors

arXiv preprint arXiv:2607.28560, Jul 2026

Abs arXiv PDF Code Website

Pretraining navigation diffusion policies rely on large-scale expert demonstrations. These data are typically generated by a fully-informed oracle planner suited to a single nominal robot. This limits the policy’s generalization to diverse embodiments and challenging scenarios (e.g., escaping dead ends or detouring long obstacles) that demand diverse local reactive behaviors with only onboard local observations. Post-training the policy with reinforcement learning (RL) offers a principled remedy. However, previous RL for diffusion approaches lead to only marginal improvements. This is because the intractable likelihood of diffusion policies renders policy gradients unstable in addition to inefficient policy exploration. To address these challenges, we propose a data-efficient diffusion RL post-training framework - GQRM (Group Q-score Reweighted Matching). Our framework introduces two complementary designs: (i) a self-bootstrapped exploration strategy with behavior perturbation that preserves the pretrained policy prior, and (ii) a group Q-score normalization mechanism that computes per-trajectory values on each state for efficient reweighted score matching. By conducting distributed online RL training across heterogeneous embodiments, the resulting fine-tuned policy, X-NavDP, achieves state-of-the-art cross-embodiment visual navigation performance, improving the overall success rate from 61.20% to 84.28% in simulation and 10% to 65% in real-world hard cases. The code and model are publicly available at https://yty-sky.github.io/x-navdp-project-page.
Cortex: A Bidirectionally Aligned Embodied Agent Framework for Long-horizon Manipulation

Jiaqi Peng, Xiqian Yu, Delin Feng, and 10 more authors

arXiv preprint arXiv:2607.05377, Jul 2026

Abs arXiv PDF Code Slides Website

While recent Vision-Language-Action (VLA) models show promise toward generalist manipulation policies, they struggle with long-horizon tasks due to their Markovian nature-relying solely on current observations. Hierarchical dual-system methods address this but suffer from a gap between high-level planning semantics and low-level execution kinematics. We introduce Cortex, a bidirectionally aligned embodied agent framework with a customized planning interface that conveys executable and tractable subtask plans from high-level VLM to low-level VLA. Specifically, we standardize manipulation subtasks into 32 canonical skill primitives and inject tractability principles, such as representative object attributes and improved trajectory reachability, into the data generation pipeline. This enables automatic annotation of over 4k hours of open-source video data and generation of 30 hours of simulation data. We further devise an event-balanced sampling strategy to construct training data for fine-tuning the framework to better handle planning ambiguity during subtask transitions, enhanced by carefully designed harness engineering from task contexts to skill constraints during inference. Both open-loop VLM and closed-loop system evaluations demonstrate Cortex’s efficacy, e.g., it outperforms monolithic baselines by 3.1% on Libero-long and 4.1% on RoboTwin. Notably, Cortex’s generalist VLM enables zero-shot completion of unseen real-world long-horizon tasks, such as multi-stage chemistry experiments, by simply combining with a fine-tuned VLA-a capability infeasible through VLA fine-tuning alone.
InternVLA-A1.5: Unifying Understanding, Latent Foresight, and Action for Compositional Generalization

Haoxiang Ma, Junhao Cai, Xiaoxu Xu, and 26 more authors

arXiv preprint arXiv:2607.04988, Jul 2026

Abs arXiv PDF Code Model Dataset Website

Unified models for robot manipulation aim to equip one policy with both the semantic priors of pretrained VLMs and the physical dynamics learned through future prediction. In practice, existing designs tend to erode the semantics of the pretrained backbone, suffer interference among heterogeneous objectives, and learn future prediction from scratch in pixel space, leaving the dynamics priors of pretrained video generators unexploited. We present InternVLA-A1.5, which builds the policy on a native VLM backbone that keeps training on VQA and subtask prediction, and attaches a lightweight unified expert for continuous action generation. Future prediction is recast as a latent-querying problem, where a small set of learnable foresight tokens condenses the task-relevant future into a compact latent code under the supervision of a frozen pretrained video generation model, so the policy inherits world-model dynamics priors without ever learning pixel-level generation. The video branch is discarded at inference, keeping real-time control. Pretrained on 1.2M robot episodes and 3M multimodal samples, InternVLA-A1.5 achieves the best overall results on all six simulation benchmarks. In the real world, the preserved semantics deliver the strongest compositional generalization on held-out instruction bindings, and the two designs together sustain long-horizon execution.
From Instruction Following to Cognitive Navigation: A Survey on the Evolution of Vision-and-Language Navigation

Kailin Lyu, Kangyi Wu, Pengna Li, and 54 more authors

CSPaper OpenPrint, Jun 2026

Abs DOI PDF Code Website

Vision-and-Language Navigation (VLN) requires embodied agents to ground natural language instructions in visual perception and make navigation decisions in complex 3D environments, making it a central problem in embodied artificial intelligence. Since the introduction of the Room-to-Room (R2R) benchmark, VLN has made substantial progress. In recent years, as research settings have gradually expanded from closed and single indoor benchmark scenarios to open-world environments, the field has undergone a profound paradigm shift from passive instruction following on fixed benchmarks to autonomous cognitive navigation in open-world settings. However, existing surveys mainly organize prior work according to technical taxonomies, lacking a systematic characterization of this paradigm evolution. To address this gap, this survey proposes an evolution-centered unified analytical framework that reviews contemporary VLN research across four progressive layers: perception, cognition, learning, and generalization. It reveals the intrinsic connections and evolutionary logic among different technical lines, identifies key open challenges at each dimension, and outlines future research directions. This survey aims to provide VLN researchers with a clear panoramic view of capability evolution, while offering the broader embodied intelligence community a systematic roadmap from closed-benchmark evaluation toward trustworthy open-world deployment.
EventVLA: Event-Driven Visual Evidence Memory for Long-Horizon Vision-Language-Action Policies

Ganlin Yang, Zhangzheng Tu, Yuqiang Yang, and 10 more authors

arXiv preprint arXiv:2606.20092, Jun 2026

Abs arXiv PDF Code Dataset Website

Memory remains a critical bottleneck for long-horizon robotic manipulation, as standard Vision-Language-Action (VLA) policies often fail when task-relevant cues become occluded or unobservable over time. While existing memory-augmented methods utilize historical context, they either suffer from severe information bottlenecks, incur high latency via decoupled dual systems, or rely on unselective buffers that accumulate massive visual redundancies. To address these limitations, we introduce EventVLA, an end-to-end framework founded on the concept of sparse visual evidence memory that comprises two core components: foundational visual anchors to retain initial and short-term contexts, and a dynamic Keyframe Evidence Memory (KEM) module. Specifically, KEM directly predicts future keyframe probabilities from the VLA’s latent embeddings to autonomously capture and store sparse, task-critical visual events. This foresight-driven mechanism empowers the policy to dynamically evaluate the future causal utility of current observations, preserving transient visual evidence before it becomes unobservable. Furthermore, we propose RoboTwin-MeM, a diagnostic benchmark specifically designed to evaluate non-Markovian manipulation tasks with interactive visual evidence. Extensive evaluations show that across 17 memory-requiring simulation tasks and 4 real-world bimanual tasks, EventVLA achieves an average success rate improvement of +40% over state-of-the-art memory-augmented VLAs.

2025

LoGoPlanner: Localization Grounded Navigation Policy with Metric-aware Visual Geometry

Jiaqi Peng, Wenzhe Cai, Yuqiang Yang, and 3 more authors

🌠Oral 2026 IEEE International Conference on Robotics & Automation, Dec 2025

Abs arXiv PDF Code Website

Trajectory planning in unstructured environments is a fundamental and challenging capability for mobile robots. Traditional modular pipelines suffer from latency and cascading errors across perception, localization, mapping, and planning modules. Recent end-to-end learning methods map raw visual observations directly to control signals or trajectories, promising greater performance and efficiency in open-world settings. However, most prior end-to-end approaches still rely on separate localization modules that depend on accurate sensor extrinsic calibration for self-state estimation, thereby limiting generalization across embodiments and environments. We introduce LoGoPlanner, a localization-grounded, end-to-end navigation framework that addresses these limitations by: (1) finetuning a long-horizon visual-geometry backbone to ground predictions with absolute metric scale, thereby providing implicit state estimation for accurate localization; (2) reconstructing surrounding scene geometry from historical observations to supply dense, fine-grained environmental awareness for reliable obstacle avoidance; and (3) conditioning the policy on implicit geometry bootstrapped by the aforementioned auxiliary tasks, thereby reducing error propagation. We evaluate LoGoPlanner in both simulation and real-world settings, where its fully end-to-end design reduces cumulative error while metric-aware geometry memory enhances planning consistency and obstacle avoidance, leading to more than a 27.3% improvement over oracle-localization baselines and strong generalization across embodiments and environments.
Ground Slow, Move Fast: A Dual-System Foundation Model for Generalizable Vision-Language Navigation

Meng Wei, Chenyang Wan, Jiaqi Peng, and 8 more authors

2026 International Conference on Learning Representations, Dec 2025

Abs arXiv PDF Code Website

While recent large vision-language models (VLMs) have improved generalization in vision-language navigation (VLN), existing methods typically rely on end-to-end pipelines that map vision-language inputs directly to short-horizon discrete actions. Such designs often produce fragmented motions, incur high latency, and struggle with real-world challenges like dynamic obstacle avoidance. We propose DualVLN, the first dual-system VLN foundation model that synergistically integrates high-level reasoning with low-level action execution. System 2, a VLM-based global planner, “grounds slowly” by predicting mid-term waypoint goals via image-grounded reasoning. System 1, a lightweight, multi-modal conditioning Diffusion Transformer policy, “moves fast” by leveraging both explicit pixel goals and latent features from System 2 to generate smooth and accurate trajectories. The dual-system design enables robust real-time control and adaptive local decision-making in complex, dynamic environments. By decoupling training, the VLM retains its generalization, while System 1 achieves interpretable and effective local navigation. DualVLN outperforms prior methods across all VLN benchmarks and real-world experiments demonstrate robust long-horizon planning and real-time adaptability in dynamic environments.
InternVLA-N1: An Open Dual-System Vision-Language Navigation Foundation Model with Learned Latent Plans

Intern Robotics, Shanghai AI Laboratory

2026 World Artificial Intelligence Conference (WAIC), Sep 2025

Abs PDF Code Model Dataset Website

We introduce InternVLA-N1, the first open dual-system vision-language navigation foundation model. Unlike previous navigation foundation models that can only take short-term actions from a limited discrete space, InternVLA-N1 decouples the task as pixel-goal planning with System 2 and agile execution with System 1. A curriculum two-stage training paradigm is devised for this framework: First, two systems are pretrained with explicit pixel goals as supervision or condition. Subsequently, we freeze System 2 and finetune the newly added latent plans with System 1 in an asynchronous end-to-end manner. Such a paradigm relying on latent plans as the intermediate representation removes the ambiguity of pixel goal planning and provides new potentials for pretraining extensions with video prediction. To enable scalable training, we develop an efficient navigation data generation pipeline and introduce InternData-N1, the largest navigation dataset to date. InternData-N1 comprises over 50 million egocentric images collected from more than 3,000 scenes, amounting to 4,839 kilometers of robot navigation experience. We evaluate InternVLA-N1 across 6 challenging navigation benchmarks, where it consistently achieves state-of-the-art performance, with improvements ranging from 3% to 28%. In particular, it demonstrates synergistic integration of long-horizon planning (>150m) and real-time decision-making (>30Hz) capabilities and can be zero-shot generalized across diverse embodiments (wheeled, quadruped, humanoid) and in-the-wild environments. All code, models, and datasets are publicly available.
NavDP: Learning Sim-to-Real Navigation Diffusion Policy with Privileged Information Guidance

Wenzhe Cai, Jiaqi Peng, Yuqiang Yang, and 6 more authors

2026 IEEE International Conference on Robotics & Automation, May 2025

Abs arXiv PDF Code Website

We present a sim-to-real navigation diffusion policy that can achieve cross-embodiment generalization in dynamic, cluttered and diverse real-world scenarios.
Towards Latency-Aware 3D Streaming Perception for Autonomous Driving

Jiaqi Peng, Tai Wang, Jiangmiao Pang, and 1 more author

2025 IEEE International Conference on Robotics and Automation (ICRA), Apr 2025

Abs arXiv PDF

Although existing 3D perception algorithms have demonstrated significant improvements in performance, their deployment on edge devices continues to encounter critical challenges due to substantial runtime latency. We propose a new benchmark tailored for online evaluation by considering runtime latency. Based on the benchmark, we build a Latency-Aware 3D Streaming Perception (LASP) framework that addresses the latency issue through two primary components: 1) latency-aware history integration, which extends query propagation into a continuous process, ensuring the integration of historical feature regardless of varying latency; 2) latency-aware predictive detection, a module that compensates the detection results with the predicted trajectory and the posterior accessed latency. By incorporating the latency-aware mechanism, our method shows generalization across various latency levels, achieving an online performance that closely aligns with 80% of its offline evaluation on the Jetson AGX Orin without any acceleration techniques.

2023

Gigavision Challenges 2022 | GigaDetection First Place Winner

Jiaqi Peng

CICAI2023, Apr 2023

PDF

2022

结合时空一致性的 FairMOT 跟踪算法优化

彭嘉淇, 王涛, 陈柯安, and 1 more author

中国图象图形学报, Apr 2022

Abs Website

Objective Video-based multiple object tracking is one of the essential tasks in computer vision like automatic driving and intelligent video surveillance system.Most of the multiple object tracking methods tend to obtain object detection results first.The integrated strategies are used to link detection bounding boxes and form object trajectories.Current object detection contexts have been developing recently.But,the challenging inconsistency issues are required to be resolved in multiple object tracking,which affected the multi-objects tracking accuracy.The multi-objects tracking inconsistency can be classified into three types as mentioned below:1) the inconsistency between the centers of the object bounding boxes and those object identity features.Many multiple object tracking methods are extracted the object re-identification (ReID) features at the object bounding boxes centers and these features are used to in associate with objects.However,those oriented ReID features are incapable to reflect the appearance of objects accurately due to the occlusion.The offsets are appeared between the best ReID feature extraction positions and bounding box centers.Current feature extraction strategy will lead to the spatial consistency problem.2) The inconsistency of the object center response between consecutive frames.Some objects can be detected and tracked in the contexted frames due to the occlusion in videos.It causes consecutive frames loss and the inconsistency between the object-center-responsed heatmaps of two consecutive frames.3) The inconsistency of the similarity assessment in the training process and testing process.Most of association step is considered as a classification problem and the cross entropy loss is used to train the model while the inter-object relations are ignored in the testing process.The feature cosine similarities of each pair of objects are used to associate them.To improve the accuracy of tracking,we facilitate a multiple object tracking method based on consistency optimization.