Img

Workshop on Collecting and Generating Resources for
Chatbots and Conversational Agents - Development and Evaluation

Shared Task Description

As part of RE-WOCHAT program activities, a shared task on data collection and annotation will be conducted. In this task, participants will generate human-machine and human-human dialogues and annotate them. Human-machine dialogues will be generated by using different online and offline chat engines, and annotations are to be generated following the provided set of guidelines. The collected dataset will be made publicly available to the research community for further research and experimentation in future editions of the workshop.

Metrics and Resources for Chat-oriented Dialogue Evaluation

The Shared Task in RE-WOCHAT is part of a larger scope initiative aiming at both collecting chat-oriented dialogue data that can be made available for research purposes and developing a framework for the automatic evaluation of chat-oriented dialogue. This effort comprises three interdependent tasks:

The current Shared Task in RE-WOCHAT focuses only on Tasks 1 and 2 described above. Task 3 will be addressed in future editions of the workshop after enough annonated data has been generated to make feasible the use of machine learning approaches.

Ways of Participation and Registration

There are four different ways of participating in the shared task:

You can register for participating in one or more of the roles described above by using this form.

Available Chatbots

These are the chatbots currently available for the shared tak. They are available in either online or stand-alone modalities, or both. If you are interested in accessing one or more of these chatbots to contribute to the data generation phase of the shared task, please register here as "data generator".

System

Modality

Information

Brief Description

JOKER

Online

Paper #396

JOKER is an example-based system that uses a database of semantically indexed dialogue examples to manage dialogue

IRIS

Online

Demo Paper

IRIS (Informal Response Interactive System) is a chat-oriented dialogue system based on the vector space model framework
pyEliza Stand-alone Website
pyElizaChatbotClient is a Python-based stand-alone version of the famous Eliza chatbot created by Weizenbaum in 1966

SARAH Online Website Sarah is a version of Alice bot, developed by Dr.Wallace in 1995
TickTock Both Paper
TickTock is a chatbot with a goal to engage users in a everyday conversation. It is a keyword based retrieval system with engagement conversational strategies

Report Format and Submissions

A short report will be required to accompany all participating chatbot systems and generated data subsets. Participating teams must complete their two-page reports according to the following Report Template no later than March 21. Report submissions must be done in electronic format through the following website https://www.softconf.com/lrec2016/RE-WOCHAT/. All submitted reports will be presented at a Shared Task Poster Session the day of the workshop.

Important Dates

Share your LRs! Initiative and ISLRN Number

Describing your LRs in the LRE Map is now a normal practice in the submission procedure of LREC (introduced in 2010 and adopted by other conferences). To continue the efforts initiated at LREC 2014 about "Sharing LRs" (data, tools, web-services, etc.), authors will have the possibility, when submitting a paper, to upload LRs in a special LREC repository. This effort of sharing LRs, linked to the LRE Map for their description, may become a new "regular" feature for conferences in our field, thus contributing to creating a common repository where everyone can deposit and share data.

As scientific work requires accurate citations of referenced work so as to allow the community to understand the whole context and also replicate the experiments conducted by other researchers, LREC 2016 endorses the need to uniquely Identify LRs through the use of the International Standard Language Resource Number (ISLRN, www.islrn.org), a Persistent Unique Identifier to be assigned to each Language Resource. The assignment of ISLRNs to LRs cited in LREC papers will be offered at submission time.