• DSTC6            >
• Breakdown detection

Task Description

The task of dialogue breakdown detection is to detect whether the system utterance causes dialogue breakdown (a situation in a dialogue where users cannot proceed with the conversation) in a given dialogue context. The participants of this track will develop a dialogue breakdown detector that outputs a dialogue breakdown label (B: breakdown, PB: possible breakdown, or NB:Not a breakdown) and a distribution of these labels.

Provided Data for the Challenge

Chat-oriented dialogues (dialogue history between a user and a chat-oriented dialogue system) are annotated with breakdown labels for each turn by muliple annotators (typically 30 annotators) as shown in the example.

See the dataset section for the developement/evaluation data we provide for the challenge. You can also download the data from this link. English and Japanese dialogues are available.

NEW! DBDC3 dataset (including test data) now available!

Evaluation Metrics

* Classification-related metrics:
Accuracy, prevision, recall and F-measure of estimated labels. These metrics are calculated by comparing the output of the detector and the gold label determined by majority voting of results of several annotators

* Distribution-related metrics:
JS divergence and mean squared error to the distribution of annotated labels. This metrics are calculated by comparing the predicted distribution of the labels with the distribution of annotation.

See the evaluation metrics section for the details of the evaluation matrics.

Getting started

You can have a look at this page to get started on dialogue breakdown detection challenge! This page shows how to use the baseline and evaluate dialogue breakdown detection results.

Getting prepared for the formal-run

You can refer to this page to get information about how to submit your runs at the formal-run.

Schedule

* Jun 1 2017: Training Data Release * Aug 28 2017 Sept 8 2017 (extended): The Deadline of Registration * Sep 17 2017: Additional Development Data Release * Sep 25 2017 Oct 9 2017 (extended): Test Data Release * Oct 8 2017 Oct 13 Oct 14 2017 (extended): The Deadline of Results Submission * Oct 31 2017: The Deadline of Paper Submission Sponsors We gratefully acknowledge the generous support provided by the following: This track is endorsed by Special Interest Group on Spoken Language Understanding and Dialogue Processing (SIG-SLUD) of the Japanese Society of Artificial Intelligence (JSAI). Task Organizers * Ryuichiro Higashinaka (NTT) * Kotaro Funakoshi (Kyoto University/Honda Research Institute Japan) * Michimasa Inaba (Hiroshima City University) * Yuiko Tsunomori (NTT Docomo) * Tetsuro Takahashi (Fujitsu) * Nobuhiro Kaji (Yahoo Japan Corporation) Contact us by: dbdc3-organizersgooglegroups.com