Intelligent Information Processing and Advanced Control Software Technology Research Team

State Key Laboratory of Intelligent Technology and Systems, Department of Computer Science and Technology, Tsinghua University

Github Contact Us

Main Research Direction

What we are doing

Multi-modal Intelligent Information Processing

Emotion recognition, sentiment analysis and intention recognition based on multi-modal information such as text, audio, video (picture).

Key Technologies of Intelligent Mobile Robots

Human-machine dialogue technology, intelligent mobile robot control technology and scene applications (business intelligent service robot, dual-mode intelligent disinfection robot).

Research on Intelligent Optimization Method

Evolution-based optimization theory, (high-dimensional, sparse, expensive) multi-objective optimization problem.

Team Members

Who are doing the work

Supervisor

Avatar

Hua Xu

Tenured Associate Professor, Associate Editor of Expert Systems with Application, Ph.D Supervisor

Avatar

Junhui Deng

Professor, Master Supervisor

Avatar

Xiaomin Sun

Associate Professor, Master Supervisor

Ph.D Student

Avatar

Ziqi Yuan

Ph.D Student

Multimodal Machine Learning

Avatar

Hanlei Zhang

Ph.D Student

Artificial Intelligence, Natrual Language Processing, Multimodal Language Understanding

Avatar

Qianrui Zhou

Ph.D Student

Artificial Intelligence, Natrual Language Processing, Multimodal Language Understanding

Avatar

Huigen Ye

Ph.D Student

Artificial Intelligence, Single-objective Optimization, Large-scale Integer Programming

Avatar

Tianxing Yang

Ph.D Student

Machine Learning for Combinatorial Optimization, Data Generation and Synthesis

Avatar

Yuetian Zou

Ph.D Student

Artificial Intelligence, Natrual Language Processing, Multimodal Language Understanding

Postgraduate Student

Avatar

Runmin Cao

Postgraduate Student

Multimodal Machine Learning, Multitask Learning

Visting Postgraduate Student

Avatar

Jianhua Su

Visting Postgraduate Student

Artificial Intelligence, Multimodal Learning, Intent understanding, Natrual Language Processing

Avatar

Weilong Liu

Visting Postgraduate Student

Artificial Intelligence, Natrual Language Processing, Multimodal Learning, Representation Learning

Avatar

Songze Li

Visting Postgraduate Student

Artificial Intelligence, Natrual Language Processing, Multimodal Language Understanding

Avatar

Shihao Zhou

Visting Postgraduate Student

Artificial Intelligence, Natrual Language Processing, Multimodal Language Understanding

Avatar

Yifan Wang

Visting Postgraduate Student

Artificial Intelligence, Natrual Language Processing, Multimodal Learning

Avatar

Xiaohan Zhang

Visting Postgraduate Student

Artificial Intelligence, Natrual Language Processing, Multimodal video understanding

Graduated

Avatar

Xin Wang

Visting Postgraduate Student

Multimoadal Learning, Facial Expression Recognition, Multi-task learning, Active Learning

Avatar

JingLiang Fang

Visting Postgraduate Student

Artificial Intelligence, Facial Expression Recognition, Machine Reading Comprehension, Natrual Language Processing

Avatar

Baozheng Zhang

Visting Postgraduate Student

Multimoadal Learning, Facial Expression Recognition, Multi-task learning, Active Learning

Avatar

Lunsong Huang

Postgraduate Student

Artificial Intelligence, Natrual Language Processing, Calibration

Avatar

Roberto Evans

Postgraduate Student

Multimodal Intent Recognition

Avatar

Huisheng Mao

Postgraduate Student

Multimoadal Learning, Facial Expression Recognition, Multi-task learning, Active Learning

Avatar

Xiaofei Chen

Visting Postgraduate Student

Artificial Intelligence, Multimodal Sentiment Analysis

Avatar

Yihe Liu

Visting Postgraduate Student

Artificial Intelligence, Natrual Language Processing, Multimodal Learning, Representation Learning

Avatar

Shaojie Zhao

Visting Postgraduate Student

Artificial Intelligence, Natrual Language Processing, Dialogue Intention Detection

Avatar

WeiYa Wang

Postgraduate Student

Artificial Intelligence, Multimodal Dialogue Intention Understanding

Avatar

Zhijing Wu

Ph.D Student

Artificial Intelligence, Natrual Intelligence, Intelligent Question-answering, Machine Reading Comprehension

Avatar

Hongyan Wang

Ph.D Student

Artificial Intelligence, Black-box Optimization, Multi-objective Optimization, Bayesian Optimization

Avatar

Huadong Li

Master’s Degree

Avatar

Kang Zhao

Visting Postgraduate Student

Avatar

Xiaoteng Li

Visting Postgraduate Student

Avatar

Jiangong Yang

Visting Postgraduate Student

Avatar

Congfeng Yin

Master’s Degree

Yuan Yuan

Ph.D

Wei Wan

Master’s Degree

Yun Wen

Master’s Degree

Bo Wang

Master’s Degree

Jia Li

Master’s Degree

Xingwei He

Master’s Degree

Jiyun Zou

Visting Postgraduate Student

Wenmeng Yu

Master’s Degree

Tingen Lin

Master’s Degree

Kaicheng Yang

Visting Postgraduate Student

Yuxiang Xie

Visiting Postgraduate Student

Tianqi Wu

Master’s Degree

Congcong Yang

Visiting Postgraduate Student

Textbooks

Textbooks that we published

Monographs

Multi-modal Sentiment Analysis

The natural interaction ability between human and machine mainly involves human-machine dialogue ability, multi-modal sentiment analysis ability, human-machine cooperation ability, and so on. To enable intelligent computers to have multi-modal sentiment analysis ability, it is necessary to equip them with a strong multi-modal sentiment analysis ability during the process of human-computer interaction. This is one of the key technologies for efficient and intelligent human-computer interaction. This book focuses on the research and practical applications of multi-modal sentiment analysis for human-computer natural interaction, particularly in the areas of multi-modal information feature representation, feature fusion, and sentiment classification. Multi-modal sentiment analysis for natural interaction is a comprehensive research field that involves the integration of natural language processing, computer vision, machine learning, pattern recognition, algorithm, robot intelligent system, human-computer interaction, etc. Currently, research on multi-modal sentiment analysis in natural interaction is developing rapidly. This book can be used as a professional textbook in the fields of natural interaction, intelligent question answering (customer service), natural language processing, human-computer interaction, etc. It can also serve as an important reference book for the development of systems and products in intelligent robots, natural language processing, human-computer interaction, and related fields. The full codes are available for use at this link.

Intent Recognition for Human-Machine Interactions

The natural interaction ability between human and machine mainly involves humanmachine dialogue ability, multi-modal sentiment analysis ability, human-machine cooperation ability, and so on. In order to realize the efficient dialogue ability of intelligent computer, it is necessary to make the computer own strong user intention understanding ability in the process of human-computer interaction. This is one of the key technologies to realize efficient and intelligent human-computer dialogue. Currently, the understanding of the objects to be analyzed requires different levels of ability such as recognition, cognition, and reasoning. The current research on human-computer interaction intention understanding is still focused on the level of recognition. The research and application of human-computer natural interaction intention recognition mainly includes the following levels: intention classification, unknown intention detection, and open intention discovery. Intention understanding for natural interaction is a comprehensive research field involving the integration of natural language processing, machine learning, algorithms, human-computer interaction, and other aspects. In recent years, our research team from State Key Laboratory of Intelligent Technology and Systems, Department of Computer Science and Technology, Tsinghua University has conducted a lot of pioneering research and applied work, which have been carried out in the field of intention understanding for natural interaction, especially in the field of intention classification, unknown intention detection and open intention discovery based on text information of human-machine dialogue based on deep learning models. Related achievements have also been published in the top academic international conferences in the field of artificial intelligence in recent years, such as ACL, AAAI, ACM MM, and well-known international journals, such as Pattern Recognition and Knowledge-based Systems. In order to systematically present the latest achievements in intention classification, unknown intention detection, and open intention discovery in academia in recent years, the relevant work achievements are systematically sorted out and presented to readers in the form of a complete systematic discussion. Currently, the research on intention understanding in natural interaction develops quickly. The author’s research team will timely sort out and summarize the latest achievements and share them with readers in the form of a series of books in the future. This book can not only be used as a professional textbook in the fields of natural interaction, intelligent question answering (customer service), natural language processing, human-computer interaction, etc., but also as an important reference book for the research and development of systems and products in intelligent robots, natural language processing, human-computer interaction, etc. As the natural interaction is a new and rapidly developing research field, limited by the author’s knowledge and cognitive scope, mistakes and shortcomings in the book are inevitable. We sincerely hope that you can give us valuable comments and suggestions for our book. Please contact ( xuhua@tsinghua.edu.cn) or a third party in the open source system platform https://thuiar.github.io/ to give us a message. All of the related source codes and datasets for this book have also been shared on the following websites https://github.com/thuiar/Books . The research work and writing of this book were supported by the National Natural Science Foundation of China (Project No. 62173195). We deeply appreciate the following student from State Key laboratory of Intelligent Technology and Systems, Department of Computer Science and Technology, Tsinghua University for her hard preparing work: Xiaofei Chen. We also deeply appreciate the following students for the related research directions of cooperative innovation work: Ting-en Lin, Hanlei Zhang, Wenmeng Yu, and Xin Wang. Without the efforts of the members of our team, the book could not be presented in a structured form in front of every reader.

Natural Interaction for Tri-Co Robots (2) Sentiment Analysis of Multimodal Interaction Information

The natural interaction ability between human and machine mainly involves human-machine dialogue ability, multi-modal sentiment analysis ability, human-machine cooperation ability and so on. In order to realize the multi-modal sentiment analysis ability of intelligent computer, it is necessary to make the computer own strong multi-modal sentiment analysis ability in the process of human-computer interaction. This is one of the key technologies to realize efficient and intelligent human-computer interaction. The research and practical application of multi-modal sentiment analysis oriented to human-computer natural interaction, this book mainly discusses the following levels of hot research content: Multi-modal Information Feature Representation, Feature Fusion and Sentiment Classification. Multi-modal sentiment analysis oriented to natural interaction is a comprehensive research field involving the integration of natural language processing, computer vision, machine learning, pattern recognition, algorithm, robot intelligent system, human-computer interaction, etc. In recent years, our research team from State Key Laboratory of Intelligent Technology and Systems, Department of Computer Science, Tsinghua University, has conducted a lot of pioneering research and applied work, which have been carried out in the field of multi-modal sentiment analysis for natural interaction, especially in the field of sentiment feature representation, feature fusion, robust sentiment analysis based on deep learning model. Related achievements have also been published in the top academic international conferences in the field of artificial intelligence in recent years, such as ACL, AAAI, ACM MM, COLING and well-known international journals, such as Pattern Recognition, Knowledge based Systems, IEEE Intelligent Systems and Expert Systems with Applications. In order to systematically present the latest achievements in multi-modal sentiment analysis in academia in recent years, the relevant work achievements are systematically sorted out and presented to readers in the form of a complete systematic discussion. Currently, the research on multi-modal sentiment analysis in natural interaction develops fastly. The author’s research team will timely sort out and summarize the latest achievements and share them with readers in the form of a series of books currenlty. This book can not only be used as a professional textbook in the fields of natural interaction, intelligent question answering (customer service), natural language processing, human-computer interaction, etc., but also as an important reference book for the research and development of systems and products in intelligent robots, natural language processing, human-computer interaction, etc. As the natural interaction is a new and rapidly developing research field, limited by the author’s knowledge and cognitive scope, mistakes and shortcomings in the book are inevitable. We sincerely hope that you can give us valuable comments and suggestions for our book. Please contact ( xuhua@tsinghua.edu.cn) or a third party in the open source system platform https://thuiar.github.io/ to give us a message. All of the related source codes and datasets for this book have also been shared on the following websites https://github.com/thuiar/Books . The research work and writing of this book were supported by the National Natural Science Foundation of China (Project No. 62173195). We deeply appreciate the following students from State Key laboratory of Intelligent Technology and Systems, Department of Computer Science, Tsinghua University for their hard preparing work: Xiaofei Chen, Yuanzhe Qiu and Jiayu Huang. We also deeply appreciate the following students for the related research directions of cooperative innovation work: Zhongwu Zhai, Wenmeng Yu, Kaicheng Yang, Jiyun Zou, Ziqi Yuan, Huisheng Mao, Wei Li,Baozheng Zhang and Yihe Liu . Without the efforts of the members of our team, the book could not be presented in a structured form in front of every reader. The full codes are available for use at this link.

Multi-modal Sentiment Analysis (Traditional Chinese Version)

This book systematically explores several hot research topics from shallow to deep, including feature representation of multi-modal emotional information, feature fusion, and emotional classification of multi-modal interaction information. The analysis of emotional information of natural interaction multi-modalities involves a comprehensive research field integrating natural language processing, computer vision, machine learning, pattern recognition, algorithms, robot intelligent systems, human-machine interaction, and other aspects. In recent years, the research team of the State Key Laboratory of Intelligent Technology and Systems in the Department of Computer Science and Technology at Tsinghua University, where the author is located, has carried out a considerable amount of innovative research and application work on the natural interaction of co-robots' multi-modal information and emotional analysis, especially in areas such as deep learning-based facial emotion feature recognition, learning representation of multi-modal emotional information, fusion of multi-modal emotional features, and the robustness of multi-modal emotional analysis under modal information missing conditions. Related achievements have also been published in top international conferences in the field of artificial intelligence in recent years, such as ACL, AAAI, ACM MM, and well-known international journals such as Pattern Recognition, Knowledge Based Systems, and Expert Systems with Applications. In order to systematically present the latest achievements of the academic community and the author's team in the field of multi-modal emotional analysis of co-robot natural interaction in recent years, this book systematically organizes the content of relevant work achievements and presents them to readers in a complete and systematic manner.

Accomplishments

Awards that we got

Chinese Association for Simulation Science and Technology Award

Intelligent Simulation Optimization Theory and Methods Based on Knowledge Mining and Collaborative Learning

First Prize of Science and Technology Invention Award of China Federation of Logistics and Purchasing

Logistics Multi-Objective Optimal Scheduling Technology and Application Based on Mobile Internet O2O Mode

Beijing Science and Technology Award Second Prize

Award-winning project: Magnetron sputtering equipment research and development and industrialization, completed by: Beijing North Microelectronics Base Equipment Technology Research Center Co., Ltd., Institute of Microelectronics, Chinese Academy of Sciences, Tsinghua University, Fudan University

Beijing Science and Technology Award Third Prize

Integrated Circuit Manufacturing Equipment Standardized Cluster Control and Complete Machine Testing Platform

First Prize of Science and Technology Progress Award of China Federation of Logistics and Purchasing

Intelligent public service platform for international transportation information

Chongqing Science and Technology Award Third Prize

Research on Key Technologies of Etching Process Control for Very Large Scale Integrated Circuits

PAKDD 2011 Best Paper

Constrained LDA for Grouping Product Features in Opinion Mining

National Science and Technology Progress Award Second Prize

Award-winning Project: 100nm High-density Plasma Etching Machine Research and Development and Industrialization (Winning Number: J-237-2-02), Grade: national level, Completion Unit: Beijing North Microelectronics Technology Research Center Co., Ltd., Chinese Academy of Sciences Microelectronics Research Institute, Tsinghua University, Peking University (collective award; ranked 1st in Tsinghua University)

Beijing Science and Technology Award First Prize

100nm High-density Plasma Etching Machine Research and Development and Industrialization

Contact Us

Get in touch with us