Utilities for downloading and building data. These can be replaced if your particular file system does not support them.

parlai.core.build_data.built(path, version_string=None)

Checks if ‘.built’ flag has been set for that task. If a version_string is provided, this has to match, or the version is regarded as not built.

parlai.core.build_data.mark_done(path, version_string=None)

Marks the path as done by adding a ‘.built’ file with the current timestamp plus a version description string if specified.

parlai.core.build_data.log_progress(curr, total, width=40)

Displays a bar showing the current progress., path, fname, redownload=False)

Downloads file using requests. If redownload is set to false, then will not download tar file again if it is present (default True).


Makes the directory and any nonexistent parent directories.

parlai.core.build_data.move(path1, path2)

Renames the given file.


Removes the given directory, if it exists.

parlai.core.build_data.untar(path, fname, deleteTar=True)

Unpacks the given archive file to the same directory, then (by default) deletes the archive file.

parlai.core.build_data.download_from_google_drive(gd_id, destination)

Uses the requests package to download a file from Google Drive.