Command Line Usage¶
This contains the command line usage for each of the standard scripts we
release. These are each included in parlai/scripts, and all may
be invoked with the “parlai” supercommand.
The parlai supercommand may be invoked from the command line by running
parlai after installing ParlAI. Its default output looks like this:
usage: parlai [-h] [--helpall] COMMAND ...
_
/")
//)
==//'=== ParlAI
/
optional arguments:
-h, --help show this help message and exit
--helpall show all commands, including advanced ones.
Commands:
display_data (dd) Display data from a task
display_model (dm) Display model predictions.
eval_model (em, eval) Evaluate a model
train_model (tm, train) Train a model
interactive (i) Interactive chat with a model on the command line
safe_interactive Like interactive, but adds a safety filter
self_chat Generate self-chats of a model
The remainder of this page describes each of the commands, their possible arguments,
and some examples of their usage.
display_data¶
Short description: Display data from a task
Aliases: dd
Basic example which iterates through the tasks specified and prints them out. Used for
verification of data loading and iteration.
For example, to make sure that bAbI task 1 (1k exs) loads one can run and to see a few of them:
Examples¶
parlai display_data --task babi:task1k:1
CLI Arguments¶
Argument |
Description |
|---|---|
|
Path to json file of options. Note: Further Command-line arguments override file-based options. |
|
Warn instead of raising if an argument passed in with –init-opt is not in the target opt. |
|
ParlAI task(s), e.g. “babi:Task1” or “babi,cbt” |
|
Choose from: train, train:ordered, valid, test. to stream data add “:stream” to any option (e.g., train:stream). by default train is random with replacement, valid is ordered, test is ordered. |
|
Batch size for minibatch training schemes |
|
Use dynamic batching |
|
Print all messages |
|
Enables some debug behavior |
|
Path to datasets, defaults to {parlai_dir}/data |
|
The model class name. can match parlai/agents/ |
|
Model file name for loading and saving models |
|
Initialize model weights and dict from this file |
|
Default: |
|
Default: |
|
Display these fields when verbose is off (e.g., “–display-add-fields label_candidates,beam_texts”) |
|
Default: |
display_model¶
Short description: Display model predictions.
Aliases: dm
Basic example which iterates through the tasks specified and runs the given model on
them.
Examples¶
parlai display_model --task babi:task1k:1 --model repeat_label
parlai display_model --task convai2 --model-file "/path/to/model_file" --datatype test
CLI Arguments¶
Argument |
Description |
|---|---|
|
Path to json file of options. Note: Further Command-line arguments override file-based options. |
|
Warn instead of raising if an argument passed in with –init-opt is not in the target opt. |
|
ParlAI task(s), e.g. “babi:Task1” or “babi,cbt” |
|
Choose from: train, train:ordered, valid, test. to stream data add “:stream” to any option (e.g., train:stream). by default train is random with replacement, valid is ordered, test is ordered. |
|
Batch size for minibatch training schemes |
|
Use dynamic batching |
|
Print all messages |
|
Enables some debug behavior |
|
Path to datasets, defaults to {parlai_dir}/data |
|
The model class name. can match parlai/agents/ |
|
Model file name for loading and saving models |
|
Initialize model weights and dict from this file |
|
Default: |
|
Display these fields when verbose is off (e.g., “–display-add-fields label_candidates,beam_texts”) |
eval_model¶
Short description: Evaluate a model
Aliases: em, eval
Basic example which iterates through the tasks specified and evaluates the given model
on them.
Examples¶
parlai eval_model --task "babi:Task1k:2" -m "repeat_label"
parlai eval_model --task convai2 --model-file "/path/to/model_file"
CLI Arguments¶
Argument |
Description |
|---|---|
|
Path to json file of options. Note: Further Command-line arguments override file-based options. |
|
Warn instead of raising if an argument passed in with –init-opt is not in the target opt. |
|
ParlAI task(s), e.g. “babi:Task1” or “babi,cbt” |
|
Choose from: train, train:ordered, valid, test. to stream data add “:stream” to any option (e.g., train:stream). by default train is random with replacement, valid is ordered, test is ordered. |
|
Batch size for minibatch training schemes |
|
Use dynamic batching |
|
Print all messages |
|
Enables some debug behavior |
|
Path to datasets, defaults to {parlai_dir}/data |
|
The model class name. can match parlai/agents/ |
|
Model file name for loading and saving models |
|
Initialize model weights and dict from this file |
|
Saves a json file of the evaluation report either as an extension to the model-file (if begins with a “.”) or a whole file path. Set to the empty string to not save at all. |
|
Saves a jsonl file of the world logs.Set to the empty string to not save at all. |
|
Choices: |
|
A positive number indicates to calculate the area under the roc curve and it also determines how many decimal digits of the predictions to keep (higher numbers->more precise); also used to determine whether or not to calculate the AUC metric |
|
The name(s) of the class to calculate the auc for |
|
Default: |
|
|
|
Default: |
|
List of metrics to show/compute, e.g. all, default,or give a list split by , like ppl,f1,accuracy,hits@1,rouge,bleuthe rouge metrics will be computed as rouge-1, rouge-2 and rouge-l |
|
Report micro-averaged metrics instead of macro averaged metrics. |
|
Fields to keep when logging. Should be a comma separated list |
|
Tensorboard logging of metrics |
|
Tensorboard logging directory, defaults to model_file.tensorboard |
generate_model_card¶
Short description: Evaluate a model
Aliases: gmc
Script to generate the model card automatically.
CLI Arguments¶
Argument |
Description |
|---|---|
|
Path to json file of options. Note: Further Command-line arguments override file-based options. |
|
Warn instead of raising if an argument passed in with –init-opt is not in the target opt. |
|
ParlAI task(s), e.g. “babi:Task1” or “babi,cbt” |
|
Choose from: train, train:ordered, valid, test. to stream data add “:stream” to any option (e.g., train:stream). by default train is random with replacement, valid is ordered, test is ordered. |
|
Batch size for minibatch training schemes |
|
Use dynamic batching |
|
Print all messages |
|
Path to datasets, defaults to {parlai_dir}/data |
|
The model class name. can match parlai/agents/ |
|
Model file name for loading and saving models |
|
Initialize model weights and dict from this file |
|
Saves a json file of the evaluation report either as an extension to the model-file (if begins with a “.”) or a whole file path. Set to the empty string to not save at all. |
|
Saves a jsonl file of the world logs.Set to the empty string to not save at all. |
|
Choices: |
|
A positive number indicates to calculate the area under the roc curve and it also determines how many decimal digits of the predictions to keep (higher numbers->more precise); also used to determine whether or not to calculate the AUC metric |
|
The name(s) of the class to calculate the auc for |
|
|
|
List of metrics to show/compute, e.g. all, default,or give a list split by , like ppl,f1,accuracy,hits@1,rouge,bleuthe rouge metrics will be computed as rouge-1, rouge-2 and rouge-l |
|
Report micro-averaged metrics instead of macro averaged metrics. |
|
Fields to keep when logging. Should be a comma separated list |
|
Tensorboard logging of metrics |
|
Tensorboard logging directory, defaults to model_file.tensorboard |
|
Default: |
|
Default: |
|
Use teacher (agent 0) or model (agent 1) |
|
New lines treat substrings as separate utterances. |
|
Ignore tokens containings these substrings (comma-separated) |
|
Path to pre-trained tokenizer vocab |
|
Path to pre-trained tokenizer merge |
|
Use BPE dropout during training. |
|
Registered name of model wrapper |
|
Where to write logs of model outputs |
|
Which tests to run; by default, run all. If |
|
Use in DEBUG mode |
|
Type of model |
|
Folder to save the model card and related contents (ie. graphs) |
|
Task to use for valid/test (defaults to the one used for training) |
|
Possible modes: gen (generation), editing, final. |
In addition, for gen mode, we can also add the following to specify which exact reports to run: data_stats, eval, safety, sample, and quant) |
|
For instance, –mode gen:data_stats:eval |
|
|
Whether or not to ignore the fromfile, jsonfile, etc. tasks if the task can be found; by default, we will (so True). |
|
Evaluation report file |
|
Path to .json file with extra arguments used for different stages of report generation and later for quantitative analyses section generation; please do NOT use the shortened format (ie. t= |
|
Quantitative report file (with different subgroups); if multiple, please separate with comma, and (optional) also add a field in the report file stating what kind of subgroup it is; note that this is only applicable for classifier type models |
|
Whether to include the miscellaneous dropdown (fields that were not included in other dropdowns); by default, the value is True. |
|
Other metrics to include in the quantitative analysis |
interactive¶
Short description: Interactive chat with a model on the command line
Aliases: i
Basic script which allows local human keyboard input to talk to a trained model.
Examples¶
parlai interactive --model-file "zoo:tutorial_transformer_generator/model"
When prompted, enter something like: Bob is Blue.\nWhat is Bob?
Input is often model or task specific. Some tasks will automatically format
The input with context for the task, e.g. -t convai2 will automatically add
personas.
CLI Arguments¶
Argument |
Description |
|---|---|
|
Path to json file of options. Note: Further Command-line arguments override file-based options. |
|
Warn instead of raising if an argument passed in with –init-opt is not in the target opt. |
|
ParlAI task(s), e.g. “babi:Task1” or “babi,cbt” |
|
Choose from: train, train:ordered, valid, test. to stream data add “:stream” to any option (e.g., train:stream). by default train is random with replacement, valid is ordered, test is ordered. |
|
Batch size for minibatch training schemes |
|
Use dynamic batching |
|
Print all messages |
|
Enables some debug behavior |
|
Path to datasets, defaults to {parlai_dir}/data |
|
The model class name. can match parlai/agents/ |
|
Model file name for loading and saving models |
|
Initialize model weights and dict from this file |
|
|
|
Set to use a prettytable when displaying examples with text candidates |
|
Display these fields when verbose is off (e.g., “–display-add-fields label_candidates,beam_texts”) |
|
Create interactive version of task |
|
Saves a jsonl file containing all of the task examples and model replies. Set to the empty string to not save at all |
|
Format to save logs in. conversations is a jsonl format, parlai is a text format. |
|
File of label_candidates to send to other agent |
|
If on, assumes single turn episodes. |
|
Fields to keep when logging. Should be a comma separated list |
safe_interactive¶
Short description: Like interactive, but adds a safety filter
Script for safety protected interaction between a local human keyboard input and a trained model.
CLI Arguments¶
Argument |
Description |
|---|---|
|
Path to json file of options. Note: Further Command-line arguments override file-based options. |
|
Warn instead of raising if an argument passed in with –init-opt is not in the target opt. |
|
ParlAI task(s), e.g. “babi:Task1” or “babi,cbt” |
|
Choose from: train, train:ordered, valid, test. to stream data add “:stream” to any option (e.g., train:stream). by default train is random with replacement, valid is ordered, test is ordered. |
|
Batch size for minibatch training schemes |
|
Use dynamic batching |
|
Print all messages |
|
Enables some debug behavior |
|
Path to datasets, defaults to {parlai_dir}/data |
|
The model class name. can match parlai/agents/ |
|
Model file name for loading and saving models |
|
Initialize model weights and dict from this file |
|
|
|
Set to use a prettytable when displaying examples with text candidates |
|
Display these fields when verbose is off (e.g., “–display-add-fields label_candidates,beam_texts”) |
|
Create interactive version of task |
|
Apply safety filtering to messages |
|
File of label_candidates to send to other agent |
|
If on, assumes single turn episodes. |
self_chat¶
Short description: Generate self-chats of a model
Allows a model to self-chat on a given task.
CLI Arguments¶
Argument |
Description |
|---|---|
|
Path to json file of options. Note: Further Command-line arguments override file-based options. |
|
Warn instead of raising if an argument passed in with –init-opt is not in the target opt. |
|
ParlAI task(s), e.g. “babi:Task1” or “babi,cbt” |
|
Choose from: train, train:ordered, valid, test. to stream data add “:stream” to any option (e.g., train:stream). by default train is random with replacement, valid is ordered, test is ordered. |
|
Batch size for minibatch training schemes |
|
Use dynamic batching |
|
Print all messages |
|
Enables some debug behavior |
|
Path to datasets, defaults to {parlai_dir}/data |
|
The model class name. can match parlai/agents/ |
|
Model file name for loading and saving models |
|
Initialize model weights and dict from this file |
|
Default: |
|
Default: |
|
Display these fields when verbose is off (e.g., “–display-add-fields label_candidates,beam_texts”) |
|
Create a self chat version of the task |
|
Number of self chats to run |
|
The number of dialogue turns before self chat ends |
|
Automatically seed conversation with messages from task dataset. |
|
If specified, loads newline-separated strings from the file as conversation starters. |
|
File to save self chat logs |
|
Format to save logs in. conversations is a jsonl format, parlai is a text format. |
|
Define a different partner for self chat |
|
Path to file containing opts to override for partner |
|
Fields to keep when logging. Should be a comma separated list |
tod_world_script¶
Short description: World for chatting with the TOD conversation structure
Base script for running TOD model-model chats.
For example, to extract gold ground truth data from the holdout version of Google SGD, run
python -u -m parlai.scripts.tod_world_script --api-schema-grounding-model parlai.tasks.google_sgd_simulation_splits.agents:OutDomainApiSchemaAgent --goal-grounding-model parlai.tasks.google_sgd_simulation_splits.agents:OutDomainGoalAgent --user-model parlai.tasks.google_sgd_simulation_splits.agents:OutDomainUserUttAgent --system-model parlai.tasks.google_sgd_simulation_splits.agents:OutDomainApiCallAndSysUttAgent --api-resp-model parlai.tasks.google_sgd_simulation_splits.agents:OutDomainApiResponseAgent -dt valid --num-episodes -1 --episodes-randomization-seed 42 --world-logs gold-valid
This file handles
Script param setup, including that used for loading agents which may have their own parameters
Running the world (including handling batching, until num episodes or length of epoch has been met).
File I/O for both reports (for metrics) and conversation logs + logic for displaying prints
CLI Arguments¶
Argument |
Description |
|---|---|
|
Path to json file of options. Note: Further Command-line arguments override file-based options. |
|
Warn instead of raising if an argument passed in with –init-opt is not in the target opt. |
|
ParlAI task(s), e.g. “babi:Task1” or “babi,cbt” |
|
Choose from: train, train:ordered, valid, test. to stream data add “:stream” to any option (e.g., train:stream). by default train is random with replacement, valid is ordered, test is ordered. |
|
Batch size for minibatch training schemes |
|
Use dynamic batching |
|
Print all messages |
|
Enables some debug behavior |
|
Path to datasets, defaults to {parlai_dir}/data |
|
Saves a json file of the evaluation report either as an extension to the model-file (if begins with a “.”) or a whole file path. Set to the empty string to not save at all. |
|
Saves a jsonl file containing all of the task examples and model replies. |
|
Choices: |
|
Number of episodes to display. Set to -1 for infinity or the number of examples of the first agent with a non-unlimited number of episodes in the world. |
|
|
|
Default: |
|
Fields to keep when logging. Should be a comma separated list |
|
The max number of full turns before chat ends, excluding prompting |
|
Define the system model for the chat. Exactly one of this or system-model must be specified |
|
Define the system agent for the chat. Exactly one of this or system-model-file must be specified |
|
Define the user model for the chat. Exactly one of this user-model must be specified. Currently assumed to be the API Call creation agent as well. |
|
Define the user agent for the chat. Exactly one of this or user-model-file must be specified. Currently assumed to be the API Call creation agent as well. |
|
Agent used for defining API response values |
|
Agent used in first turn to grounding api call/response agents with api schemas. Will use EmptyApiSchemaAgent if both this and |
|
Agent used in first turn to grounding user agent with goal. Will use EmptyGoalAgent if not set |
|
If set and |
train_model¶
Short description: Train a model
Aliases: tm, train
Training script for ParlAI.
The standard way to train a model. After training, also computes validation and test error.
The user must provide a model (with --model) and a task (with
--task).
Examples¶
parlai train_model --model ir_baseline --task dialog_babi:Task:1 --model-file /tmp/model
parlai train_model --model seq2seq --task babi:Task10k:1 --model-file '/tmp/model' --batchsize 32 --learningrate 0.5
CLI Arguments¶
Argument |
Description |
|---|---|
|
Path to json file of options. Note: Further Command-line arguments override file-based options. |
|
Warn instead of raising if an argument passed in with –init-opt is not in the target opt. |
|
ParlAI task(s), e.g. “babi:Task1” or “babi,cbt” |
|
Choose from: train, train:ordered, valid, test. to stream data add “:stream” to any option (e.g., train:stream). by default train is random with replacement, valid is ordered, test is ordered. |
|
Batch size for minibatch training schemes |
|
Use dynamic batching |
|
Print all messages |
|
Enables some debug behavior |
|
Path to datasets, defaults to {parlai_dir}/data |
|
The model class name. can match parlai/agents/ |
|
Model file name for loading and saving models |
|
Initialize model weights and dict from this file |
|
Task to use for valid/test (defaults to the one used for training) |
|
A ‘.opt’ file that is used for final eval. Useful for setting skip-generation to false. ‘datatype’ must be included as part of the opt. |
|
Set dynamic batching at evaluation time. Set to off for train-only dynamic batching. Set to none (default) to use same setting as –dynamic-batching. |
|
Number of background workers (training only) |
|
Default: |
|
Default: |
|
End training after n model updates |
|
Log every n training steps |
|
Validate every n seconds. Saves model to model_file (if set) whenever best val metric is found |
|
Validate every n training steps. Saves model to model_file (if set) whenever best val metric is found |
|
Saves the model to model_file.checkpoint after every n seconds (default -1, never). |
|
Saves the model to model_file.checkpoint after every validation (default |
|
Validate every n epochs. Saves model to model_file (if set) whenever best val metric is found |
|
Number of iterations of validation where result does not improve before we stop training |
|
Key into report table for selecting best validation |
|
The direction in which to optimize the validation metric, i.e. maximize or minimize |
|
List of metrics to show/compute, e.g. all, default,or give a list split by , like ppl,f1,accuracy,hits@1,rouge,bleuthe rouge metrics will be computed as rouge-1, rouge-2 and rouge-l |
|
Report micro-averaged metrics instead of macro averaged metrics. |
|
Saves a jsonl file of the world logs.Set to the empty string to not save at all. |
|
Choices: |
|
|
|
Fields to keep when logging. Should be a comma separated list |
|
Tensorboard logging of metrics |
|
Tensorboard logging directory, defaults to model_file.tensorboard |
|
Enable W&B logging of metrics |
|
W&B project name. Defaults to timestamp. Usually the name of the sweep. |
|
W&B entity name. |
|
Enable logging of model artifacts to weight and biases |
|
Creates a ClearML Task. Default: False. If True, ClearML logging will be enabled. |
|
ClearML Project Name. All the logs will be stored under this project in ClearML WebUI. If not set, default will set to ParlAI. |
|
ClearML Task Name. All the logs will be stored under this task in ClearML WebUI. If not set, default will set to “Default Task”. |
|
Path to pre-trained tokenizer vocab |
|
Path to pre-trained tokenizer merge |
|
Use BPE dropout during training. |