BERT Classifier

This directory contains an implementations of a classifier based on a pretrained language model BERT (Devlin et al. https://arxiv.org/abs/1810.04805). It relies on the pytorch implementation provided by Hugging Face (https://github.com/huggingface/pytorch-pretrained-BERT).

Basic Examples

Train a classifier on the SNLI tas.

parlai train_model -m bert_classifier -t snli --classes 'entailment' 'contradiction' 'neutral' -mf /tmp/BERT_snli -bs 20

In the example above, tokenized input sentence will look as following: [CLS] premise : motor ##cy ##cl ##ists racing on a track . hypothesis : people are racing . [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD]

BertClassifierAgent Options

TorchAgent Arguments

Argument	Description
--interactive-mode, --i	Whether in full interactive mode or not, which means generating text or retrieving from a full set of candidates, which is necessary to actually do full dialogue. However, during training or quick validation (e.g. PPL for generation or ranking a few candidates for ranking models) you might want these set to off. Typically, scripts can set their preferred default behavior at the start, e.g. eval scripts. Default: False.
--embedding-type, --emb	Choose between different strategies for initializing word embeddings. Default is random, but can also preinitialize from Glove or Fasttext. Preinitialized embeddings can also be fixed so they are not updated during training. Choices: random, glove, glove-fixed, fasttext, fasttext-fixed, fasttext_cc, fasttext_cc-fixed. Default: random.
--embedding-projection, --embp	If pretrained embeddings have a different dimensionality than your embedding size, strategy for projecting to the correct size. If the dimensions are the same, this is ignored unless you append “-force” to your choice. Default: random.
--fp16	Use fp16 computations. Default: False.
--fp16-impl	Implementation of FP16 to use Choices: safe, mem_efficient. Default: safe.
--rank-candidates, --rc	Whether the model should parse candidates for ranking. Default: False.
--truncate, --tr	Truncate input lengths to increase speed / use less memory. Default: -1.
--text-truncate	Text input truncation length: if not specified, this will default to truncate
--label-truncate	Label truncation length: if not specified, this will default to truncate
--history-reversed	Reverse the history Default: False.
--history-size, --histsz	Number of past dialog utterances to remember. Default: -1.
--person-tokens, --pt	Add person tokens to history. adds p1 in front of input text and p2 in front of past labels when available or past utterances generated by the model. these are added to the dictionary during initialization. Default: False.
--split-lines	Split the dialogue history on newlines and save in separate vectors Default: False.
--delimiter	Join history lines with this token, defaults to newline Default: \n.
--special-tok-lst	Comma separated list of special tokens. In case of ambiguous parses from special tokens, the ordering provided in this arg sets precedence.
-gpu, --gpu	Which GPU to use Default: -1.
--no-cuda	Disable GPUs even if available. otherwise, will use GPUs if available on the device. Default: False.

Optimizer Arguments

Argument	Description
--optimizer, --opt	Optimizer choice. Possible values: adadelta, adagrad, adam, adamw, sparseadam, adamax, asgd, sgd, radam, rprop, rmsprop, optimizer, nadam, lbfgs, mem_eff_adam, adafactor. Choices: adadelta, adagrad, adam, adamw, sparseadam, adamax, asgd, sgd, radam, rprop, rmsprop, optimizer, nadam, lbfgs, mem_eff_adam, adafactor. Default: sgd.
--learningrate, --lr	Learning rate Default: 1.
--gradient-clip, --clip	Gradient clipping using l2 norm Default: 0.1.
--adafactor-eps	Epsilon values for adafactor optimizer: regularization constants for square gradient and parameter scale respectively Default: 1e-30,1e-3. Recommended: 1e-30,1e-3.
--momentum, --mom	If applicable, momentum value for optimizer. Default: 0.
--nesterov	If applicable, whether to use nesterov momentum. Default: True.
--nus, --nu	If applicable, nu value(s) for optimizer. can use a single value like 0.7 or a comma-separated tuple like 0.7,1.0 Default: 0.7.
--betas, --beta	If applicable, beta value(s) for optimizer. can use a single value like 0.9 or a comma-separated tuple like 0.9,0.999 Default: 0.9,0.999.
--weight-decay, --wdecay	Weight decay on the weights.

BPEHelper Arguments

Argument	Description
--bpe-vocab	Path to pre-trained tokenizer vocab
--bpe-merge	Path to pre-trained tokenizer merge
--bpe-dropout	Use BPE dropout during training.

Learning Rate Scheduler

Argument	Description
--lr-scheduler	Learning rate scheduler. Choices: reduceonplateau, none, fixed, invsqrt, cosine, linear. Default: reduceonplateau.
--lr-scheduler-patience	LR scheduler patience. In number of validation runs. If using fixed scheduler, LR is decayed every validations. Default: 3.
--lr-scheduler-decay	Decay factor for LR scheduler, or how much LR is multiplied by when it is lowered. Default: 0.5.
--invsqrt-lr-decay-gamma	Constant used only to find the lr multiplier for the invsqrt scheduler. Must be set for –lr-scheduler invsqrt Default: -1.

Torch Classifier Arguments

Argument	Description
--classes	The name of the classes.
--class-weights	Weight of each of the classes for the softmax
--threshold	During evaluation, threshold for choosing ref class; only applies to binary classification Default: 0.5.
--print-scores	Print probability of chosen class during interactive mode Default: False.
--data-parallel	Uses nn.DataParallel for multi GPU Default: False.
--classes-from-file	Loads the list of classes from a file
--ignore-labels	Ignore labels provided to model
--update-classifier-head-only	Freeze the encoder and update the classifier head only Default: False.

BERT Classifier Arguments

Argument	Description
--add-cls-token	Add [CLS] token to text vec Default: True.
--sep-last-utt	Separate the last utterance into a differentsegment with [SEP] token in between Default: False.
--classifier-layers	List of classifier layers comma-separated with layer’s dimension where applicable. For example: linear,64 linear,32 relu

BertDictionaryAgent Options

BPEHelper Arguments

Argument	Description
--bpe-vocab	Path to pre-trained tokenizer vocab
--bpe-merge	Path to pre-trained tokenizer merge
--bpe-dropout	Use BPE dropout during training.