Docs

mage Wizard of Wikipedia: Knowledge-Powered Conversational Agents

The Wizard of Wikipedia is an open-domain dialogue task for training agents that can converse knowledgably about open-domain topics! A detailed description may be found in Dinan et al. (ICLR 2019).

Abstract

In open-domain dialogue intelligent agents should exhibit the use of knowledge, however there are few convincing demonstrations of this to date. The most popular sequence to sequence models typically "generate and hope" generic utterances that can be memorized in the weights of the model when mapping from input utterance(s) to output, rather than employing recalled knowledge as context. Use of knowledge has so far proved difficult, in part because of the lack of a supervised learning benchmark task which exhibits knowledgeable open dialogue with clear grounding. To that end we collect and release a large dataset with conversations directly grounded with knowledge retrieved from Wikipedia. We then design architectures capable of retrieving knowledge, reading and conditioning on it, and finally generating natural responses. Our best performing dialogue models are able to conduct knowledgeable discussions on open-domain topics as evaluated by automatic metrics and human evaluations, while our new benchmark allows for measuring further improvements in this important research direction.

Datasets

You can train your own ParlAI agent on the Wizard of Wikipedia task with -t wizard_of_wikipedia. See the ParlAI quickstart for help.

The ParlAI MTurk collection scripts are also made available, for those interested in replication, analysis, or additional data collection. The MTurk task for evaluating pre-trained models is made available in this directory.

Leaderboard

Human Evaluations

Model Paper Seen Rating Unseen Rating
Retrieval Trans MemNet Dinan et al. (2019) 3.43 3.14
Two-stage Generative Trans MemNet Dinan et al. (2019) 2.92 2.93
Human performance Dinan et al. (2019) 4.13 4.34

Retrieval Models

Model Paper Test Seen R@1 Test Unseen R@1
Transformer MemNet (w/ pretraining) Dinan et al. (2019) 87.4 69.8
BoW Memnet Dinan et al. (2019) 71.3 33.1
IR baseline Dinan et al. (2019) 17.8 14.2
Random Dinan et al. (2019) 1.0 1.0

Generative Models

Model Paper Test Seen PPL Test Unseen PPL
End-to-end Transformer MemNet Dinan et al. (2019) 63.5 97.3
Two-Stage Transformer Memnet Dinan et al. (2019) 46.5 84.8
Vanilla Transformer (no knowledge) Dinan et al. (2019) 41.8 87.0

Pretrained models

You can evaluate the pretrained End-to-end generative models via:

python examples/eval_model.py \
    -bs 64 -t wizard_of_wikipedia:generator:random_split \
    -mf models:wizard_of_wikipedia/end2end_generator/model

This produces the following metrics:

{'f1': 0.1717, 'ppl': 61.21, 'know_acc': 0.2201, 'know_chance': 0.02625}

This differs slightly from the results in the paper, as it is a recreation trained from scratch for public release.

You can also evaluate the model on the unseen topic split too:

python examples/eval_model.py \
    -bs 64 -t wizard_of_wikipedia:generator:topic_split \
    -mf models:wizard_of_wikipedia/end2end_generator/model

This will produce:

{'f1': 0.1498, 'ppl': 103.1, 'know_acc': 0.1123, 'know_chance': 0.02496}

You can evaluate a retrieval model on the full dialogue task by running the following script:

python projects/wizard_of_wikipedia/scripts/eval_retrieval_model.py

Check back later for more pretrained models soon!

Citation

If you use the dataset or models in your own work, please cite with the following BibTex entry:

@inproceedings{dinan2019wizard,
  author={Emily Dinan and Stephen Roller and Kurt Shuster and Angela Fan and Michael Auli and Jason Weston},
  title={{W}izard of {W}ikipedia: Knowledge-powered Conversational Agents},
  booktitle = {Proceedings of the International Conference on Learning Representations (ICLR)},
  year={2019},
}