Second Workshop on Chatbots and Conversational Agent Technologies
Shared Task Description
As part of WOCHAT program activities, the Shared Task on Data Collection and Annotation will continue. In this task, participants are required to generate human-machine and human-human dialogues, as well as to produce turn-level annotations on them. Human-machine dialogues can be generated by using different online and offline chat engines made available for such purpose. Annotations are generated following the provided set of guidelines. The collected dataset will be made publicly available to the research community for further research and experimentation in future editions of the workshop.
Metrics and Resources for Chat-oriented Dialogue Evaluation
The Shared Task in WOCHAT is part of a larger scope initiative aiming at both collecting chat-oriented dialogue data that can be made available for research purposes and developing a framework for the automatic evaluation of chat-oriented dialogue. This effort comprises three interdependent tasks:
- Task 1. Chat data collection: participating teams will produce dialogues between human users and chat engines, as well as between humans only.
- Task 2. Subjective evaluation: participating teams will manually evaluate a selection of the generated dialogues according to different subjective evaluation metrics.
- Task 3. Subrogated metrics: participant teams will attempt to model the manually generated subjective evaluation metrics by using machine learning techniques.
Similar to RE-WOCHAT, the current Shared Task in WOCHAT focuses on Tasks 1 and 2 only. Task 3 will be addressed in future editions of the workshop after enough annonated data has been generated to make feasible the use of machine learning approaches.
Ways of Participation and Registration
There are four different ways of participating in the shared task:
- Chatbot provider. You own a chatbot engine and you want to provide access to it either by distributing a standalone version of it or by giving access to it via a webservice or web interface.
- Data generator. You are willing to use one or more of the provided chatbots to generate dialogue sessions with it. You can proceed to the chatbot_info page to start chatting right away!
- Data provider. You own or have access to a chatbot but you cannot provide access to it. However, you can generate dialogue sessions with it and want to share the generated data with other participants.
- Data annotator. You are willing to annonate some of the generated and/or shared dialogue sessions by following the annotation guidelines provided.
You can register for participating in one or more of the roles described above by using this form.
Report Format and Submissions
A short report will be required to accompany all participating role activities in the Shared Task.
Report submissions to WOCHAT should be in Springer LNCS format (up to 4 pages).
Paper submissions must be done in electronic format through the following
conference submission system.
All submitted reports will be presented at a Shared Task Poster Session the day of the workshop.
Important Dates
July 19, 2016: Report Submission DeadlineJuly 26, 2016: Report Acceptance NotificationAugust 2, 2016: Camera Ready Version DeadlineSeptember 20, 2016: Workshop is held at IVA 2016 in Los Angeles (California)