Ji, Heng

74 publications

TMLR 2026 A Survey of Self-Evolving Agents: What, When, How, and Where to Evolve on the Path to Artificial Super Intelligence Huan-ang Gao, Jiayi Geng, Wenyue Hua, Mengkang Hu, Xinzhe Juan, Hongzhang Liu, Shilong Liu, Jiahao Qiu, Xuan Qi, Qihan Ren, Yiran Wu, Hongru Wang, Han Xiao, Yuhang Zhou, Shaokun Zhang, Jiayi Zhang, Jinyu Xiang, Yixiong Fang, Qiwen Zhao, Dongrui Liu, Cheng Qian, Zhenhailong Wang, Minda Hu, Huazheng Wang, Qingyun Wu, Heng Ji, Mengdi Wang

ICLR 2026 BEAT: Visual Backdoor Attacks on VLM-Based Embodied Agents via Contrastive Trigger Learning Qiusi Zhan, Hyeonjeong Ha, Rui Yang, Sirui Xu, Hanyang Chen, Liangyan Gui, Yu-Xiong Wang, Huan Zhang, Heng Ji, Daniel Kang

ICLR 2026 Constructive Distortion: Improving MLLMs with Attention-Guided Image Warping Dwip Dalal, Gautam Vashishtha, Utkarsh Mishra, Jeonghwan Kim, Madhav Kanda, Hyeonjeong Ha, Svetlana Lazebnik, Heng Ji, Unnat Jain

ICLR 2026 Energy-Based Transformers Are Scalable Learners and Thinkers Alexi Gladstone, Ganesh Nanduru, Md Mofijul Islam, Peixuan Han, Hyeonjeong Ha, Aman Chadha, Yilun Du, Heng Ji, Jundong Li, Tariq Iqbal

ICLR 2026 Multimodal Policy Internalization for Conversational Agents Zhenhailong Wang, Jiateng Liu, Amin Fazel, Ritesh Sarkhel, Xing Fan, Xiang Li, Chenlei Guo, Heng Ji, Ruhi Sarikaya

ICLR 2026 Neural Synchrony Between Socially Interacting Language Models Zhining Zhang, Wentao Zhu, Chi Han, Yizhou Wang, Heng Ji

ICLR 2026 Perception-Aware Policy Optimization for Multimodal Reasoning Zhenhailong Wang, Xuehang Guo, Sofia Stoica, Haiyang Xu, Hongru Wang, Hyeonjeong Ha, Xiusi Chen, Yangyi Chen, Ming Yan, Fei Huang, Heng Ji

TMLR 2026 Prioritizing Image-Related Tokens Enhances Vision-Language Pre-Training Yangyi Chen, Hao Peng, Tong Zhang, Heng Ji

ICLR 2026 RM-R1: Reward Modeling as Reasoning Xiusi Chen, Gaotang Li, Ziqi Wang, Bowen Jin, Cheng Qian, Yu Wang, Hongru Wang, Yu Zhang, Denghui Zhang, Tong Zhang, Hanghang Tong, Heng Ji

ICLR 2026 ReIn: Conversational Error Recovery with Reasoning Inception Takyoung Kim, Jinseok Nam, Chandrayee Basu, Xing Fan, Chengyuan Ma, Heng Ji, Gokhan Tur, Dilek Hakkani-Tür

ICLR 2026 SWERank: Software Issue Localization with Code Ranking Revanth Gangi Reddy, Tarun Suresh, JaeHyeok Doo, Ye Liu, Xuan-Phi Nguyen, Yingbo Zhou, Semih Yavuz, Caiming Xiong, Heng Ji, Shafiq Joty

TMLR 2026 The Landscape of Agentic Reinforcement Learning for LLMs: A Survey Guibin Zhang, Hejia Geng, Xiaohang Yu, Zhenfei Yin, Zaibin Zhang, Zelin Tan, Heng Zhou, Zhong-Zhi Li, Xiangyuan Xue, Yijiang Li, Yifan Zhou, Yang Chen, Chen Zhang, Yutao Fan, Zihu Wang, Songtao Huang, Francisco Piedrahita Velez, Yue Liao, Hongru Wang, Mengyue Yang, Heng Ji, Jun Wang, Shuicheng Yan, Philip Torr, Lei Bai

ICLR 2026 TrustGen: A Platform of Dynamic Benchmarking on the Trustworthiness of Generative Foundation Models Yue Huang, Chujie Gao, Siyuan Wu, Haoran Wang, Xiangqi Wang, Jiayi Ye, Yujun Zhou, Yanbo Wang, Jiawen Shi, Qihui Zhang, Han Bao, Zhaoyi Liu, Yuan Li, Tianrui Guan, Peiran Wang, Haomin Zhuang, Dongping Chen, Kehan Guo, Andy Zou, Bryan Hooi, Caiming Xiong, Elias Stengel-Eskin, Hongyang Zhang, Hongzhi Yin, Huan Zhang, Huaxiu Yao, Jieyu Zhang, Jaehong Yoon, Kai Shu, Ranjay Krishna, Swabha Swayamdipta, Weijia Shi, Xiang Li, Yuexing Hao, Zhihao Jia, Zhize Li, Xiuying Chen, Zhengzhong Tu, Xiyang Hu, Tianyi Zhou, Jieyu Zhao, Lichao Sun, Furong Huang, Or Cohen-Sasson, Prasanna Sattigeri, Anka Reuel, Max Lamparth, Yue Zhao, Nouha Dziri, Yu Su, Huan Sun, Heng Ji, Chaowei Xiao, Mohit Bansal, Nitesh V Chawla, Jian Pei, Jianfeng Gao, Michael Backes, Philip S. Yu, Neil Zhenqiang Gong, Pin-Yu Chen, Bo Li, Dawn Song, Xiangliang Zhang

ICLR 2026 mCLM: A Modular Chemical Language Model That Generates Functional and Makeable Molecules Carl Edwards, Chi Han, Gawon Lee, Thao Nguyen, Sara Szymkuć, Chetan Kumar Prasad, Bowen Jin, Jiawei Han, Ying Diao, Ge Liu, Hao Peng, Bartosz Andrzej Grzybowski, Martin D. Burke, Heng Ji

FTML 2025 Artificial Intelligence for Science in Quantum, Atomistic, and Continuum Systems Xuan Zhang, Limei Wang, Jacob Helwig, Youzhi Luo, Cong Fu, Yaochen Xie, Meng Liu, Yuchao Lin, Zhao Xu, Keqiang Yan, Keir Adams, Maurice Weiler, Xiner Li, Tianfan Fu, Yucheng Wang, Alex Strasser, Haiyang Yu, Yuqing Xie, Xiang Fu, Shenglong Xu, Yi Liu, Yuanqi Du, Alexandra Saxton, Hongyi Ling, Hannah Lawrence, Hannes Stärk, Shurui Gui, Carl Edwards, Nicholas Gao, Adriana Ladera, Tailin Wu, Elyssa F. Hofgard, Aria Mansouri Tehrani, Rui Wang, Ameya Daigavane, Montgomery Bohde, Jerry Kurtin, Qian Huang, Tuong Phung, Minkai Xu, Chaitanya K. Joshi, Simon V. Mathis, Kamyar Azizzadenesheli, Ada Fang, Alán Aspuru-Guzik, Erik J. Bekkers, Michael M. Bronstein, Marinka Zitnik, Anima Anandkumar, Stefano Ermon, Pietro Liò, Rose Yu, Stephan Günnemann, Jure Leskovec, Heng Ji, Jimeng Sun, Regina Barzilay, Tommi S. Jaakkola, Connor W. Coley, Xiaoning Qian, Xiaofeng Qian, Tess E. Smidt, Shuiwang Ji

IJCAI 2025 Automating Intervention Discovery from Scientific Literature: A Progressive Ontology Prompting and Dual-LLM Framework Yuting Hu, Dancheng Liu, Qingyun Wang, Charles Yu, Chenhui Xu, Qingxiao Zheng, Heng Ji, Jinjun Xiong

ICLR 2025 CoRNStack: High-Quality Contrastive Data for Better Code Retrieval and Reranking Tarun Suresh, Revanth Gangi Reddy, Yifei Xu, Zach Nussbaum, Andriy Mulyar, Brandon Duderstadt, Heng Ji

ICML 2025 Contrastive Visual Data Augmentation Yu Zhou, Bingxuan Li, Tang Mohan, Xiaomeng Jin, Te-Lin Wu, Kuan-Hao Huang, Heng Ji, Kai-Wei Chang, Nanyun Peng

NeurIPS 2025 DyMU: Dynamic Merging and Virtual Unmerging for Efficient Variable-Length VLMs Zhenhailong Wang, Senthil Purushwalkam, Caiming Xiong, Silvio Savarese, Heng Ji, Ran Xu

ICLR 2025 Eliminating Position Bias of Language Models: A Mechanistic Approach Ziqi Wang, Hanlin Zhang, Xiner Li, Kuan-Hao Huang, Chi Han, Shuiwang Ji, Sham M. Kakade, Hao Peng, Heng Ji

ICML 2025 EmbodiedBench: Comprehensive Benchmarking Multi-Modal Large Language Models for Vision-Driven Embodied Agents Rui Yang, Hanyang Chen, Junyu Zhang, Mark Zhao, Cheng Qian, Kangrui Wang, Qineng Wang, Teja Venkat Koripella, Marziyeh Movahedi, Manling Li, Heng Ji, Huan Zhang, Tong Zhang

NeurIPS 2025 Fire360: A Benchmark for Robust Perception and Episodic Memory in Degraded 360° Firefighting Video Aditi Tiwari, Farzaneh Masoud, Dac Trong Nguyen, Jill Kraft, Heng Ji, Klara Nahrstedt

ICLR 2025 Fragment and Geometry Aware Tokenization of Molecules for Structure-Based Drug Design Using Language Models Cong Fu, Xiner Li, Blake Olson, Heng Ji, Shuiwang Ji

ICML 2025 Geometry Informed Tokenization of Molecules for Language Model Generation Xiner Li, Limei Wang, Youzhi Luo, Carl Edwards, Shurui Gui, Yuchao Lin, Heng Ji, Shuiwang Ji

ICLRW 2025 MolCap-Arena: A Comprehensive Captioning Benchmark on Language-Enhanced Molecular Property Prediction Carl Edwards, Ziqing Lu, Ehsan Hajiramezanali, Tommaso Biancalani, Heng Ji, Gabriele Scalia

ICLR 2025 OpenHands: An Open Platform for AI Software Developers as Generalist Agents Xingyao Wang, Boxuan Li, Yufan Song, Frank F. Xu, Xiangru Tang, Mingchen Zhuge, Jiayi Pan, Yueqi Song, Bowen Li, Jaskirat Singh, Hoang H. Tran, Fuqiang Li, Ren Ma, Mingzhang Zheng, Bill Qian, Yanjun Shao, Niklas Muennighoff, Yizhe Zhang, Binyuan Hui, Junyang Lin, Robert Brennan, Hao Peng, Heng Ji, Graham Neubig

NeurIPS 2025 PARTONOMY: Large Multimodal Models with Part-Level Visual Understanding Ansel Blume, Jeonghwan Kim, Hyeonjeong Ha, Elen Chatikyan, Xiaomeng Jin, Khanh Duy Nguyen, Nanyun Peng, Kai-Wei Chang, Derek Hoiem, Heng Ji

TMLR 2025 Scaling Laws for Predicting Downstream Performance in LLMs Yangyi Chen, Binxuan Huang, Yifan Gao, Zhengyang Wang, Jingfeng Yang, Heng Ji

CVPR 2025 Search and Detect: Training-Free Long Tail Object Detection via Web-Image Retrieval Mankeerat Sidhu, Hetarth Chopra, Ansel Blume, Jeonghwan Kim, Revanth Gangi Reddy, Heng Ji

ICML 2025 SyncMind: Measuring Agent Out-of-Sync Recovery in Collaborative Software Engineering Xuehang Guo, Xingyao Wang, Yangyi Chen, Sha Li, Chi Han, Manling Li, Heng Ji

NeurIPS 2025 ToolRL: Reward Is All Tool Learning Needs Cheng Qian, Emre Can Acikgoz, Qi He, Hongru Wang, Xiusi Chen, Dilek Hakkani-Tür, Gokhan Tur, Heng Ji

TMLR 2025 Towards LifeSpan Cognitive Systems Yu Wang, Chi Han, Tongtong Wu, Xiaoxin He, Wangchunshu Zhou, Nafis Sadeq, Xiusi Chen, Zexue He, Wei Wang, Gholamreza Haffari, Heng Ji, Julian McAuley

ICML 2025 Training Software Engineering Agents and Verifiers with SWE-Gym Jiayi Pan, Xingyao Wang, Graham Neubig, Navdeep Jaitly, Heng Ji, Alane Suhr, Yizhe Zhang

ICLRW 2025 Training Software Engineering Agents and Verifiers with SWE-Gym Jiayi Pan, Xingyao Wang, Graham Neubig, Navdeep Jaitly, Heng Ji, Alane Suhr, Yizhe Zhang

TMLR 2025 Understanding Emergent In-Context Learning from a Kernel Regression Perspective Chi Han, Ziqi Wang, Han Zhao, Heng Ji

NeurIPS 2025 Variational Supervised Contrastive Learning Ziwen Wang, Jiajun Fan, Thao Nguyen, Heng Ji, Ge Liu

ICCV 2025 Verbalized Representation Learning for Interpretable Few-Shot Generalization Cheng-Fu Yang, Da Yin, Wenbo Hu, Heng Ji, Nanyun Peng, Bolei Zhou, Kai-Wei Chang

TMLR 2025 Visually Descriptive Language Model for Vector Graphics Reasoning Zhenhailong Wang, Joy Hsu, Xingyao Wang, Kuan-Hao Huang, Manling Li, Jiajun Wu, Heng Ji

TMLR 2024 A Single Transformer for Scalable Vision-Language Modeling Yangyi Chen, Xingyao Wang, Hao Peng, Heng Ji

ICML 2024 CHEMREASONER: Heuristic Search over a Large Language Model’s Knowledge Space Using Quantum-Chemical Feedback Henry W. Sprueill, Carl Edwards, Khushbu Agarwal, Mariefel V Olarte, Udishnu Sanyal, Conrad Johnston, Hongbin Liu, Heng Ji, Sutanay Choudhury

ICLR 2024 CRAFT: Customizing LLMs by Creating and Retrieving from Specialized Toolsets Lifan Yuan, Yangyi Chen, Xingyao Wang, Yi Fung, Hao Peng, Heng Ji

CVPR 2024 DRESS: Instructing Large Vision-Language Models to Align and Interact with Humans via Natural Language Feedback Yangyi Chen, Karan Sikka, Michael Cogswell, Heng Ji, Ajay Divakaran

NeurIPSW 2024 Eliminating Position Bias of Language Models: A Mechanistic Approach Ziqi Wang, Hanlin Zhang, Xiner Li, Kuan-Hao Huang, Chi Han, Shuiwang Ji, Sham M. Kakade, Hao Peng, Heng Ji

ICLR 2024 Enabling Lanuguage Models to Implicitly Learn Self-Improvement Ziqi Wang, Le Hou, Tianjian Lu, Yuexin Wu, Yunxuan Li, Hongkun Yu, Heng Ji

ICML 2024 Executable Code Actions Elicit Better LLM Agents Xingyao Wang, Yangyi Chen, Lifan Yuan, Yizhe Zhang, Yunzhu Li, Hao Peng, Heng Ji

ICLRW 2024 Executable Code Actions Elicit Better LLM Agents Xingyao Wang, Yangyi Chen, Lifan Yuan, Yizhe Zhang, Yunzhu Li, Hao Peng, Heng Ji

ICLRW 2024 If LLM Is the Wizard, Then Code Is the Wand: A Survey on How Code Empowers Large Language Models to Serve as Intelligent Agents Ke Yang, Jiateng Liu, John Wu, Chaoqi Yang, Yi Fung, Sha Li, Zixuan Huang, Xu Cao, Xingyao Wang, Heng Ji, ChengXiang Zhai

NeurIPSW 2024 Infogent: An Agent-Based Framework for Web Information Aggregation Revanth Gangi Reddy, Sagnik Mukherjee, Jeonghwan Kim, Zhenhailong Wang, Dilek Hakkani Tur, Heng Ji

NeurIPS 2024 Invariant Tokenization of Crystalline Materials for Language Model Enabled Generation Keqiang Yan, Xiner Li, Hongyi Ling, Kenna Ashen, Carl Edwards, Raymundo Arróyave, Marinka Zitnik, Heng Ji, Xiaofeng Qian, Xiaoning Qian, Shuiwang Ji

ICML 2024 Iterative Preference Learning from Human Feedback: Bridging Theory and Practice for RLHF Under KL-Constraint Wei Xiong, Hanze Dong, Chenlu Ye, Ziqi Wang, Han Zhong, Heng Ji, Nan Jiang, Tong Zhang

ICLR 2024 MINT: Evaluating LLMs in Multi-Turn Interaction with Tools and Language Feedback Xingyao Wang, Zihan Wang, Jiateng Liu, Yangyi Chen, Lifan Yuan, Hao Peng, Heng Ji

ICML 2024 Position: TrustLLM: Trustworthiness in Large Language Models Yue Huang, Lichao Sun, Haoran Wang, Siyuan Wu, Qihui Zhang, Yuan Li, Chujie Gao, Yixin Huang, Wenhan Lyu, Yixuan Zhang, Xiner Li, Hanchi Sun, Zhengliang Liu, Yixin Liu, Yijue Wang, Zhikun Zhang, Bertie Vidgen, Bhavya Kailkhura, Caiming Xiong, Chaowei Xiao, Chunyuan Li, Eric P. Xing, Furong Huang, Hao Liu, Heng Ji, Hongyi Wang, Huan Zhang, Huaxiu Yao, Manolis Kellis, Marinka Zitnik, Meng Jiang, Mohit Bansal, James Zou, Jian Pei, Jian Liu, Jianfeng Gao, Jiawei Han, Jieyu Zhao, Jiliang Tang, Jindong Wang, Joaquin Vanschoren, John Mitchell, Kai Shu, Kaidi Xu, Kai-Wei Chang, Lifang He, Lifu Huang, Michael Backes, Neil Zhenqiang Gong, Philip S. Yu, Pin-Yu Chen, Quanquan Gu, Ran Xu, Rex Ying, Shuiwang Ji, Suman Jana, Tianlong Chen, Tianming Liu, Tianyi Zhou, William Yang Wang, Xiang Li, Xiangliang Zhang, Xiao Wang, Xing Xie, Xun Chen, Xuyu Wang, Yan Liu, Yanfang Ye, Yinzhi Cao, Yong Chen, Yue Zhao

ICMLW 2024 Weak-to-Strong Extrapolation Expedites Alignment Chujie Zheng, Ziqi Wang, Heng Ji, Minlie Huang, Nanyun Peng

AAAI 2023 ADEPT: A DEbiasing PrompT Framework Ke Yang, Charles Yu, Yi Ren Fung, Manling Li, Heng Ji

ICLR 2023 Augmentation with Projection: Towards an Effective and Efficient Data Augmentation Paradigm for Distillation Ziqi Wang, Yuexin Wu, Frederick Liu, Daogao Liu, Le Hou, Hongkun Yu, Jing Li, Heng Ji

ICLR 2023 Learning to Decompose Visual Features with Latent Textual Prompts Feng Wang, Manling Li, Xudong Lin, Hairong Lv, Alex Schwing, Heng Ji

ICLR 2023 Logical Entity Representation in Knowledge-Graphs for Differentiable Rule Learning Chi Han, Qizheng He, Charles Yu, Xinya Du, Hanghang Tong, Heng Ji

NeurIPS 2023 Paxion: Patching Action Knowledge in Video-Language Foundation Models Zhenhailong Wang, Ansel Blume, Sha Li, Genglin Liu, Jaemin Cho, Zineng Tang, Mohit Bansal, Heng Ji

NeurIPS 2023 Revisiting Out-of-Distribution Robustness in NLP: Benchmarks, Analysis, and LLMs Evaluations Lifan Yuan, Yangyi Chen, Ganqu Cui, Hongcheng Gao, FangYuan Zou, Xingyi Cheng, Heng Ji, Zhiyuan Liu, Maosong Sun

AAAI 2023 SumREN: Summarizing Reported Speech About Events in News Revanth Gangi Reddy, Heba Elfardy, Hou Pong Chan, Kevin Small, Heng Ji

CVPR 2023 Towards Fast Adaptation of Pretrained Contrastive Models for Multi-Channel Video-Language Retrieval Xudong Lin, Simran Tiwari, Shiyuan Huang, Manling Li, Mike Zheng Shou, Heng Ji, Shih-Fu Chang

AAAI 2023 Video Event Extraction via Tracking Visual States of Arguments Guang Yang, Manling Li, Jiajie Zhang, Xudong Lin, Heng Ji, Shih-Fu Chang

NeurIPSW 2023 What a Scientific Language Model Knows and Doesn't Know About Chemistry Lawrence Zhao, Carl Edwards, Heng Ji

CVPR 2022 CLIP-Event: Connecting Text and Images with Event Structures Manling Li, Ruochen Xu, Shuohang Wang, Luowei Zhou, Xudong Lin, Chenguang Zhu, Michael Zeng, Heng Ji, Shih-Fu Chang

ICLR 2022 Chemical-Reaction-Aware Molecule Representation Learning Hongwei Wang, Weijiang Li, Xiaomeng Jin, Kyunghyun Cho, Heng Ji, Jiawei Han, Martin D. Burke

ICLR 2022 Cross-Lingual Transfer with Class-Weighted Language-Invariant Representations Ruicheng Xian, Heng Ji, Han Zhao

NeurIPS 2022 Language Models with Image Descriptors Are Strong Few-Shot Video-Language Learners Zhenhailong Wang, Manling Li, Ruochen Xu, Luowei Zhou, Jie Lei, Xudong Lin, Shuohang Wang, Ziyi Yang, Chenguang Zhu, Derek Hoiem, Shih-Fu Chang, Mohit Bansal, Heng Ji

AAAI 2022 MuMuQA: Multimedia Multi-Hop News Question Answering via Cross-Media Knowledge Extraction and Grounding Revanth Gangi Reddy, Xilin Rui, Manling Li, Xudong Lin, Haoyang Wen, Jaemin Cho, Lifu Huang, Mohit Bansal, Avirup Sil, Shih-Fu Chang, Alexander G. Schwing, Heng Ji

AAAI 2022 Open Vocabulary Electroencephalography-to-Text Decoding and Zero-Shot Sentiment Classification Zhenhailong Wang, Heng Ji

CVPR 2021 Separating Skills and Concepts for Novel Visual Question Answering Spencer Whitehead, Hui Wu, Heng Ji, Rogerio Feris, Kate Saenko

NeurIPS 2021 Skyformer: Remodel Self-Attention with Gaussian Kernel and Nystr\"om Method Yifan Chen, Qi Zeng, Heng Ji, Yun Yang

ECCV 2020 Learning to Learn Words from Visual Scenes Dídac Surís, Dave Epstein, Heng Ji, Shih-Fu Chang, Carl Vondrick

AAAI 2015 A Novel Neural Topic Model and Its Supervised Extension Ziqiang Cao, Sujian Li, Yang Liu, Wenjie Li, Heng Ji

IJCAI 2015 Constrained Information-Theoretic Tripartite Graph Clustering to Identify Semantically Similar Relations Chenguang Wang, Yangqiu Song, Dan Roth, Chi Wang, Jiawei Han, Heng Ji, Ming Zhang