The task of dialogue breakdown detection is to detect whether the system utterance causes dialogue breakdown (a situation in a dialogue where users cannot proceed with the conversation) in a given dialogue context. The participants of this track will develop a dialogue breakdown detector that outputs a dialogue breakdown label (B: breakdown, PB: possible breakdown, or NB:Not a breakdown) and a distribution of these labels.
Chat-oriented dialogues (dialogue history between a user and a chat-oriented dialogue system) are annotated with breakdown labels for each turn by muliple annotators (typically 30 annotators) as shown in the example.
See the dataset section for the developement/evaluation data we provide for the challenge. You can also download the data from this link. English and Japanese dialogues are available.
* Classification-related metrics: Accuracy, prevision, recall and F-measure of estimated labels. These metrics are calculated by comparing the output of the detector and the gold label determined by majority voting of results of several annotators
* Distribution-related metrics: JS divergence and mean squared error to the distribution of annotated labels. This metrics are calculated by comparing the predicted distribution of the labels with the distribution of annotation.
See the evaluation metrics section for the details of the evaluation matrics.
You can have a look at this page to get started on dialogue breakdown detection challenge! This page shows how to use the baseline and evaluate dialogue breakdown detection results.
You can refer to this page to get information about how to submit your runs at the formal-run.
* Jun 1 2017: Training Data Release
This track is endorsed by Special Interest Group on Spoken Language Understanding and Dialogue Processing (SIG-SLUD) of the Japanese Society of Artificial Intelligence (JSAI).
* Ryuichiro Higashinaka (NTT) * Kotaro Funakoshi (Kyoto University/Honda Research Institute Japan) * Michimasa Inaba (Hiroshima City University) * Yuiko Tsunomori (NTT Docomo) * Tetsuro Takahashi (Fujitsu) * Nobuhiro Kaji (Yahoo Japan Corporation)
Contact us by: dbdc3-organizers