core.build_data

Utilities for downloading and building data. These can be replaced if your particular file system does not support them.

parlai.core.build_data.built(path, version_string=None)

Checks if ‘.built’ flag has been set for that task. If a version_string is provided, this has to match, or the version is regarded as not built.

parlai.core.build_data.mark_done(path, version_string=None)

Marks the path as done by adding a ‘.built’ file with the current timestamp plus a version description string if specified.

parlai.core.build_data.log_progress(curr, total, width=40)

Displays a bar showing the current progress.

parlai.core.build_data.download(url, path, fname, redownload=False)

Downloads file using requests. If redownload is set to false, then will not download tar file again if it is present (default True).

parlai.core.build_data.make_dir(path)

Makes the directory and any nonexistent parent directories.

parlai.core.build_data.move(path1, path2)

Renames the given file.

parlai.core.build_data.remove_dir(path)

Removes the given directory, if it exists.

parlai.core.build_data.untar(path, fname, deleteTar=True)

Unpacks the given archive file to the same directory, then (by default) deletes the archive file.

parlai.core.build_data.download_from_google_drive(gd_id, destination)

Uses the requests package to download a file from Google Drive.