A post-processing method for detecting unknown intent of dialogue system via pre-trained deep neural network classifier

Abstract

With the maturity and popularity of dialogue systems, detecting user’s unknown intent in dialogue systems has become an important task. It is also one of the most challenging tasks since we can hardly get examples, prior knowledge or the exact numbers of unknown intents. In this paper, we propose SofterMax and deep novelty detection (SMDN), a simple yet effective post-processing method for detecting unknown intent in dialogue systems based on pre-trained deep neural network classifiers. Our method can be flexibly applied on top of any classifiers trained in deep neural networks without changing the model architecture. We calibrate the confidence of the softmax outputs to compute the calibrated confidence score (i.e., SofterMax) and use it to calculate the decision boundary for unknown intent detection. Furthermore, we feed the feature representations learned by the deep neural networks into traditional novelty detection algorithm to detect unknown intents from different perspectives. Finally, we combine the methods above to perform the joint prediction. Our method classifies examples that differ from known intents as unknown and does not require any examples or prior knowledge of it. We have conducted extensive experiments on three benchmark dialogue datasets. The results show that our method can yield significant improvements compared with the state-of-the-art baselines1 1The code will be available at https://github.com/tnlin/SMDN..

Publication
Knowledge-Based Systems
Tingen Lin
Master’s Degree

Joined the team in 2017, obtained a master’s degree in 2020, and an outstanding master’s degree graduate of the Department of Computer Science,Tsinghua University.

Hua Xu
Hua Xu
Tenured Associate Professor, Associate Editor of Expert Systems with Application, Ph.D Supervisor

Related