MIPAL

Multimodal Intelligence & Physical Application Lab

Spatial Intelligence Group (SIG), Tongji University

MIPAL (Multimodal Intelligence & Physical Application Lab) is a research group within the Spatial Intelligence Group (SIG) at Tongji University, led by Dr. Xun Jiang. Our team is also working closely with Koala URAN Ltd. towards real-world applications in spatial intelligence.

Our research spans multimodal learning, spatial intelligence (particularly from an egocentric view), and VLM/LLM reasoning enhancement. We are also highly interested in embodied multimodal spatial perception and navigation.

We are currently recruiting Master's and Doctoral students for enrollment in 2027. If you are a diligent student who enjoys multimodal research and practical robotic applications, you are highly welcome to join MIPAL and grow together with us!

Multimodal Learning
Egocentric / Spatial Intelligence
VLM/LLM Reasoning
Embodied Perception & Navigation

Join Us GitHub

News

2026/07Two papers about egocentric error detection and point-cloud representation were accepted by ACM MM 2026! 🎉🎉
2026/07One paper about egocentric online action segmentation was accepted by TIP! 🎉🎉
2026/03Five papers about multimodal learning, egocentric vision, and acoustic UAV localization accepted by ICME 2026! 🎉🎉 Congratulations to all my co-authors!
2026/02Two papers about multimodal learning and multimodal acoustic field synthesis accepted by CVPR and CVPR Findings 2026!
2026/01Our work on audio and acoustic signal processing was accepted by ICASSP 2026! 🎉🎉 Congratulations to my collaborator Qichen!
2026/01Our work on generalizable egocentric task verification was accepted by IEEE TPAMI!
2025/12Our work on egocentric online action segmentation was accepted by IEEE TMM!
2025/11Two papers about multimodal learning and egocentric video understanding accepted by AAAI 2026! 🎉🎉 Congratulations to Disen and Chong!

People

Faculty & Guest Faculty

Xun Jiang

Tongji University

Web · Scholar · GitHub · Email

Jingran Zhang

Koala URAN Ltd.

Web · GitHub

Lei Wang

Tongyi Lab

Web · Scholar · GitHub

PhD Students

Qin Li

Ph.D Student (2026-)

Chen Wu

Ph.D Student (2026-)

Master’s Students

Chong Liu

Master’s Student (2024-)

Web

Shuangle Li

Master’s Student (2024-)

Web

Pengqi Chen

Master’s Student (2024-)

Web

Kaiyuan Deng

Master’s Student (2024-)

Web

Yufan Gu

Master’s Student (2025-)

Ziheng Zhang

Master’s Student (2026-)

Le Bo

Master’s Student (2026-)

Siyuan Zeng

Master’s Student (2026-)

Jiankang Zhao

Master’s Student (2026-)

Web

Visiting Members

Qichen Tan

Visiting Scholar

Web · Scholar · Email

Shenshen Li

Visiting Ph.D Student

Web · Scholar

Chen He

Visiting Ph.D Student

Simiao Wang

Undergraduate

Zeyang Liu

Undergraduate

Kaiwen Tan

Undergraduate

Applications

Quadrupedal Robot

Based on the quadruped robot platform, we conducted research on navigation, multimodal spatial memory, and object localization algorithms and employed engineering implementation, with a primary objective of addressing practical application challenges across various scenarios such as inspection, delivery, and guided tours.

AI/XR Wearable Devices

We have conducted research across egocentric vision, such as content analysis, interaction generation, and streaming memory for egocentric multimodal large models, while continuously exploring application scenarios for wearable smart devices--such as AI glasses and XR equipment, facilitating them with more active intelligence ability.

Physical Agent

We are dedicated to developing integrated hardware-software physical agents that bring digital-space agents into the real world. By constructing an agent reasoning framework infused with spatial knowledge and integrating advanced embodied capability models, we are developing a universal "embodied brain" to empower a variety of robotic platforms.

Publications

Selected Journal Papers

Egocentric Online Action Segmentation via Parametric Context Memory Learning

Xun Jiang, Xing Xu, Chong Liu, Zheng Wang, Jingkuan Song, Zhe Sun, Andrzej Cichocki, Heng Tao Shen

IEEE Transactions on Image Processing · TIP 2026

[Paper - Coming Soon] [Code]

Egocentric Vision; Temporal Action Segmentation; Parametric Memory

Generalizable Egocentric Task Verification Via Cross-Modal Hybrid Hypergraph Matching

Xun Jiang, Xing Xu, Zheng Wang, Jingkuan Song, Fumin Shen, Heng Tao Shen

IEEE Transactions on Pattern Analysis and Machine Intelligence · TPAMI 2026

[Paper] [Code]

Egocentric Vision, Cross-modal Task Verification, Hypergraph Learning

Egocentric Online Action Segmentation via Evidential Temporal Contextualization

Xun Jiang, Xing Xu, Zhangye Han, Zheng Wang, Jingkuan Song, Heng Tao Shen

IEEE Transactions on Multimedia · TMM 2026

[Paper] [Code]

Egocentric Vision; Temporal Action Segmentation; Streaming Video Understanding

Zero-Shot Video Moment Retrieval with Angular Reconstructive Text Embeddings

Xun Jiang, Xing Xu, Zailei Zhou, Yang Yang, Fumin Shen, Heng Tao Shen

IEEE Transactions on Multimedia · TMM 2024

[Paper] [Code]

Video Content Understanding; Weakly-Supervised Learning; CLIP

SDN: Semantic Decoupling Network for Temporal Language Grounding

Xun Jiang, Xing Xu, Jingran Zhang, Fumin Shen, Zuo Cao, Heng Tao Shen

IEEE Transactions on Neural Networks and Learning Systems · TNNLS 2022

[Paper] [Code]

Video Content Understanding; Vision-Language; Multimodal Learning

Selected Conference Papers

Multimodal Learning on Low-Quality Data with Conformal Predictive Self-Calibration

Xun Jiang, Yufan Gu, Disen Hu, Yuqing Hou, Yazhou Yao, Fumin Shen, Heng Tao Shen, Xing Xu

IEEE/CVF Conference on Computer Vision and Pattern Recognition · CVPR 2026

[Paper] [Code]

Multimodal Learning, Low-quality Multimodal Data, Conformal Prediction

Hyper-Opinion Vagueness Quantification for Robust Multimodal Learning

Disen Hu, Xun Jiang, Xiaofeng Cao, Zheng Wang, Jingkuan Song, Heng Tao Shen, Xing Xu

AAAI Conference on Artificial Intelligence · AAAI 2026

[Paper] [Code]

Multimodal Learning, Robust Multimodal Learning, Hyper-Opinion Vagueness

Geometric Gradient Divergence Modulation for Imbalanced Multimodal Learning

Disen Hu, Xun Jiang, Zhe Sun, Hao Yang, Chong Peng, Peng Yan, Heng Tao Shen, Xing Xu

ACM International Conference on Multimedia · ACM MM 2025

[Paper] [Code]

Multimodal Learning, Imbalanced Multimodal Learning, Hyperspace Polyhedron

Procedural Heterogeneous Graph Completion for Natural Language Task Verification in Egocentric Videos

Xun Jiang, Zhiyi Huang, Xing Xu, Jingkuan Song, Fumin Shen, Heng Tao Shen

IEEE/CVF Conference on Computer Vision and Pattern Recognition · CVPR 2025

[Paper] [Code]

Natural Language-based Egocentric Task Verification; Heterogeneous Graph Completion; Multimodal Learning; Procedural Task Understanding

Counterfactually Augmented Event Matching for De-biased Temporal Sentence Grounding

Xun Jiang, Zhuoyuan Wei, Shenshen Li, Xing Xu, Jingkuan Song, Heng Tao Shen

ACM International Conference on Multimedia · ACM MM 2024

[Paper] [Code]

Video Content Understanding, De-biased Video Grounding; Counterfactual Reasoning; Multimodal Learning

Embracing Unimodal Aleatoric Uncertainty for Robust Multimodal Fusion

Zixian Gao*, Xun Jiang*, Xing Xu, Fumin Shen, Yujie Li, Heng Tao Shen (* Equal Contribution)

IEEE/CVF Computer Vision and Pattern Recognition Conference · CVPR 2024

[Paper] [Code]

Multimodal Learning; Model Robustness; Uncertainty in Deep Learning

Joint Searching and Grounding: Multi-Granularity Video Content Retrieval

Zhiguo Chen*, Xun Jiang*, Xing Xu, Zuo Cao, Yijun Mo, Heng Tao Shen (* Equal Contribution)

ACM International Conference on Multimedia · ACM MM 2023

[Paper] [Code]

Multimedia Retrieval; Video Content Understanding; Multimodal Learning

Faster Video Moment Retrieval with Point-Level Supervision

Xun Jiang, Zailei Zhou, Xing Xu, Yang Yang, Guoqing Wang, Heng Tao Shen

ACM International Conference on Multimedia · ACM MM 2023

[Paper] [Code]

Video Content Retrieval; Point-level Supervision; Retrieval Efficiency

DHHN: Dual Hierarchical Hybrid Network for Weakly-Supervised Audio-Visual Video Parsing

Xun Jiang, Xing Xu, Zhiguo Chen, Jingran Zhang, Jingkuan Song, Fumin Shen, Huimin Lu, Heng Tao Shen

ACM International Conference on Multimedia · ACM MM 2022

[Paper] [Code]

Video Content Understanding; Action Localization; Audio-Visual Learning

Semi-Supervised Video Paragraph Grounding With Contrastive Encoder

Xun Jiang, Xing Xu, Jingran Zhang, Fumin Shen, Zuo Cao, Heng Tao Shen

IEEE/CVF Conference on Computer Vision and Pattern Recognition · CVPR 2022

[Paper]

Video Content Understanding, Semi-Supervised Learning; Multimodal Learning

Recruitment

Practical Master‘s Student

Enrollment in 2027.09

"Talk is cheap, show me your code." Practical, application-oriented students may receive a much higher special scholarship and internship recommendation. Come and connect yourself to enterprises and get your offer early!

Academic Master‘s Student

Enrollment in 2027.09

“SOTA is all you need.” You may study in a relatively free and open environment during your three-year journey--with higher demands on publications. You are a pre-PhD student, actually.

Doctoral Student

Enrollment in 2027.03 or 2027.09

"The last degree you can get." A way to an endless way with unknown risks and painful study, but also with charming scenery you may never have seen before.

Visiting Members

Academic only

"Let's be better together!"--We are always open to any sincere, diligent, and optimistic friends.

Contact

xun_jiang@tongji.edu.cn

Spatial Intelligence Group (SIG)
Tongji University
Shanghai, China