Fifth Dialog State Tracking Challenge @ SLT2016

Data Overview

General Characteristics

In this challenge, participants will use TourSG corpus to develop the components. TourSG consists of dialog sessions on touristic information for Singapore collected from Skype calls between tour guides and tourists. All the recorded dialogs with the total length of 21 hours, for English and Chinese each, have been manually transcribed and annotated with speech act and semantic labels for each turn level.

Since each subject in these dialogs tends to be expressed not just in a single turn, but through a series of multiple turns, dialog states are defined in these conversations for each sub-dialog level. A full dialog session is divided into sub-dialogs considering their topical coherence and then they are categorized by topics. Each sub-dialog assigned to one of major topic categories will have an additional frame structure with slot value pairs to represent some more details about the subject discussed within the sub-dialog.

Different from DSTC4, in which English dialogs were used, DSTC5 will focus on developing and evaluating cross-language strategies for Dialog State Tracking. In this edition of the challenge, trackers must be trained on English dialogs but will be evaluated on Chinese dialogs. A small annotated development set of Chinese dialogs will be provided too.

Regarding pilot tasks, the same cross-language approach will be evaluated. English dialogs will be used as training data and the systems will be evaluated on Chinese data. A small subset of Chinese dialogs will be provided along with the training data. For pilot tasks, annotations are provided at the utterance level and, accordingly, systems must deal with information at the utterance level. These annotations involve semantic slots, speech acts and full utterances.

Data Available for DSTC5

Train and Development data: manual transcriptions and annotations at both utterance and sub-dialog levels will be provided for 35 English dialogs and 2 Chinese dialogs for training the trackers and fine-tuning their parameters.

Test data: for the main task, manual transcriptions will be provided for 10 Chinese dialogs for evaluating the trackers. In the case of the pilot tasks, 8 Chinese dialogs will be used for evaluating each task.

Both datasets will be released free of charge to all registered challenege participants after signing a license agreement with ETPL-A*STAR. The dataset will include transcribed and annotated dialogs, as well as ontology objects describing the annotations.