Workshops and Session Series on Chatbots and Conversational Agents
Shared Task Description
As part of the WOCHAT workshop series, a Shared Task on Data Collection, Annotation and Evaluation is being conducted.
In this task, participants are required to generate human-machine and human-human dialogues, as well as
to produce turn-level annotations on them. Human-machine dialogues can be generated by using different
online and offline chat engines made available for such purpose. Annotations are generated following the
provided set of guidelines.
The collected dataset is publicly available to the research community for further
research and experimentation in future editions of the workshop. The most recent version of the dataset is available in this link.
Metrics and Resources for Chat-oriented Dialogue Evaluation
WOCHAT's Shared Task is part of a larger scope initiative aiming at both collecting chat-oriented dialogue data that can be made available for research purposes and developing a framework for the automatic evaluation of chat-oriented dialogue. This effort comprises three interdependent tasks:
- Task 1. Chat data collection: participating teams will produce dialogues between human users and chat engines, as well as between humans only.
- Task 2. Subjective evaluation: participating teams will manually evaluate a selection of the generated dialogues according to different subjective evaluation metrics.
- Task 3. Subrogated metrics: participant teams will attempt to model the manually generated subjective evaluation metrics by using machine learning techniques.
Ways of Participation and Registration
There are four different ways of participating in the shared task:
- Chatbot provider. You own a chatbot engine and you want to provide access to it either by distributing a standalone version of it or by giving access to it via a webservice or web interface.
- Data generator. You are willing to use one or more of the provided chatbots to generate dialogue sessions with it. You can proceed to the chatbot_info page to start chatting right away!
- Data provider. You own or have access to a chatbot but you cannot provide access to it. However, you can generate dialogue sessions with it and want to share the generated data with other participants.
- Data annotator. You are willing to annonate some of the generated and/or shared dialogue sessions by following the annotation guidelines provided.
You can register for participating in one or more of the roles described above by using this form.