parlai.core.metrics

Provides standard metric evaluations for dialog, as well as an aggregator.

class parlai.core.metrics.MetricDisplayData(title, description)

Bases: tuple

title

Alias for field number 0

description

Alias for field number 1

class parlai.core.metrics.Metric

Bases: abc.ABC

Base class for storing metrics.

Subclasses should define .value(). Examples are provided for each subclass.

property is_global

Indicates whether this metric should be reported globally or per-task.

property macro_average

Indicates whether this metric should be macro-averaged when globally reported.

abstract value() → float

Return the value of the metric as a float.

classmethod many(*objs: List[Union[List[Union[int, float, torch.Tensor]], torch.Tensor]]) → List[parlai.core.metrics.Metric]

Construct many of a Metric from the base parts.

Useful if you separately compute numerators and denominators, etc.

classmethod from_mask(metric_per_token: torch.Tensor, mask: torch.Tensor) → List[parlai.core.metrics.Metric]

From token-level metrics, returns an aggregate MyMetric per example in the batch.

Parameters

• metric_per_token – a (batchsize x num_tokens) Tensor

• mask – a (batchsize x num_tokens) Tensor to mask out tokens that should not be considered in the aggregate metric calculation.

Returns

a (batchsize) Tensor

class parlai.core.metrics.FixedMetric(value: Union[int, float, torch.Tensor])

Bases: parlai.core.metrics.Metric

Fixed metrics are verified to be the same when combined, or throw an error.

FixedMetric is used for things like total_train_updates, which should not be combined across different multitasks or different workers.

__init__(value: Union[int, float, torch.Tensor])

Initialize self. See help(type(self)) for accurate signature.

value() → float

Return the value of the metric as a float.

class parlai.core.metrics.SumMetric(sum_: Union[int, float, torch.Tensor] = 0)

Bases: parlai.core.metrics.Metric

Class that keeps a running sum of some metric.

Examples of SumMetric include things like “exs”, the number of examples seen since the last report, which depends exactly on a teacher.

__init__(sum_: Union[int, float, torch.Tensor] = 0)

Initialize self. See help(type(self)) for accurate signature.

value() → float

Return the value of the metric as a float.

class parlai.core.metrics.AverageMetric(numer: Union[int, float, torch.Tensor], denom: Union[int, float, torch.Tensor] = 1)

Bases: parlai.core.metrics.Metric

Class that keeps a running average of some metric.

Examples of AverageMetrics include hits@1, F1, accuracy, etc. These metrics all have per-example values that can be directly mapped back to a teacher.

property macro_average

Indicates whether this metric should be macro-averaged when globally reported.

__init__(numer: Union[int, float, torch.Tensor], denom: Union[int, float, torch.Tensor] = 1)

Initialize self. See help(type(self)) for accurate signature.

value() → float

Return the value of the metric as a float.

class parlai.core.metrics.MacroAverageMetric(metrics: Dict[str, parlai.core.metrics.Metric])

Bases: parlai.core.metrics.Metric

Class that represents the macro average of several numbers.

Used for aggregating task level metrics. It is only used for things that are AverageMetrics already.

__init__(metrics: Dict[str, parlai.core.metrics.Metric]) → None

Initialize self. See help(type(self)) for accurate signature.

value() → float

Return the value of the metric as a float.

class parlai.core.metrics.TimerMetric(value: Union[int, float, torch.Tensor], start_time: Optional[float] = None, end_time: Optional[float] = None)

Bases: parlai.core.metrics.Metric

A timer metric keep tracks of the first/last times it was used.

__init__(value: Union[int, float, torch.Tensor], start_time: Optional[float] = None, end_time: Optional[float] = None)

Initialize self. See help(type(self)) for accurate signature.

value() → float

Return the value of the metric as a float.

class parlai.core.metrics.GlobalMetric

Bases: object

A global metric is one that should not be aggregated across different tasks.

Examples of global metric include things like learning rate and updates. These need to be accumulated or averaged over multiple parleys, but cannot be correlated with a single task.

Key to it is the notion that any one worker or any one task already has a global view of the value, and so no combinations should be done. Note this is different then a FixedMetric, in that a GlobalMetric can be still averaged across multiple parleys(), but a FixedMetric is always fixed.

class parlai.core.metrics.GlobalFixedMetric(value: Union[int, float, torch.Tensor])

Bases: parlai.core.metrics.GlobalMetric, parlai.core.metrics.FixedMetric

Global fixed metric.

Used for things like total_train_updates.

class parlai.core.metrics.GlobalSumMetric(sum_: Union[int, float, torch.Tensor] = 0)

Bases: parlai.core.metrics.GlobalMetric, parlai.core.metrics.SumMetric

Global sum metric.

Used for ‘exs’ and ‘updates’.

class parlai.core.metrics.GlobalAverageMetric(numer: Union[int, float, torch.Tensor], denom: Union[int, float, torch.Tensor] = 1)

Bases: parlai.core.metrics.GlobalMetric, parlai.core.metrics.AverageMetric

Global Average metric.

Used for things like learning rate, and many agent-specific metrics.

class parlai.core.metrics.LegacyMetric(numer: Union[int, float, torch.Tensor], denom: Union[int, float, torch.Tensor] = 1)

Bases: parlai.core.metrics.GlobalAverageMetric

Legacy Metrics are reported by agent as float.

class parlai.core.metrics.GlobalTimerMetric(value: Union[int, float, torch.Tensor], start_time: Optional[float] = None, end_time: Optional[float] = None)

Bases: parlai.core.metrics.GlobalMetric, parlai.core.metrics.TimerMetric

class parlai.core.metrics.F1Metric(numer: Union[int, float, torch.Tensor], denom: Union[int, float, torch.Tensor] = 1)

Bases: parlai.core.metrics.AverageMetric

Helper class which computes token-level F1.

class parlai.core.metrics.ExactMatchMetric(numer: Union[int, float, torch.Tensor], denom: Union[int, float, torch.Tensor] = 1)

Bases: parlai.core.metrics.AverageMetric

class parlai.core.metrics.BleuMetric(numer: Union[int, float, torch.Tensor], denom: Union[int, float, torch.Tensor] = 1)

Bases: parlai.core.metrics.AverageMetric

static compute(guess: str, answers: List[str], k: int = 4) → Optional[parlai.core.metrics.BleuMetric]

Compute approximate BLEU score between guess and a set of answers.

class parlai.core.metrics.FairseqBleuMetric(pred: Union[torch.Tensor, List[int]], ref: Union[torch.Tensor, List[int]], pad_idx: int, eos_idx: int, unk_idx: int, order: int)

Bases: parlai.core.metrics.Metric

Re-implementation of https://github.com/pytorch/fairseq/blob/main/fairseq/scoring/bleu.py.

__init__(pred: Union[torch.Tensor, List[int]], ref: Union[torch.Tensor, List[int]], pad_idx: int, eos_idx: int, unk_idx: int, order: int)

Initialize self. See help(type(self)) for accurate signature.

property macro_average

Indicates whether this metric should be macro-averaged when globally reported.

value() → float

Reimplementation of Fairseq’s score.

static compute_many(guess: torch.Tensor, answers: torch.Tensor, pad_idx, end_idx, unk_idx)

Return BLEU-1..4 using fairseq and tokens.

class parlai.core.metrics.RougeMetric(numer: Union[int, float, torch.Tensor], denom: Union[int, float, torch.Tensor] = 1)

Bases: parlai.core.metrics.AverageMetric

static compute_many(guess: str, answers: List[str], measure: str = 'r') → Tuple[Optional[parlai.core.metrics.RougeMetric], Optional[parlai.core.metrics.RougeMetric], Optional[parlai.core.metrics.RougeMetric]]

Compute ROUGE score between guess and any answer.

Done with compute_many due to increased efficiency.

Returns

(rouge-1, rouge-2, rouge-L)

class parlai.core.metrics.IntraDistinctMetric(numer: Union[int, float, torch.Tensor], denom: Union[int, float, torch.Tensor] = 1)

Bases: parlai.core.metrics.AverageMetric

Compute intra-distinct (per-utterance).

classmethod compute(text: str, ngram: int = 1)

Parameters

• text – The text to compute metric over

• ngram – n-gram length

class parlai.core.metrics.InterDistinctMetric(counts: Counter[Tuple])

Bases: parlai.core.metrics.Metric

Compute inter-distinct metric over corpus-level.

__init__(counts: Counter[Tuple])

Parameters

counts – collections.Counter of ngram -> frequency

value()

Return the value of the metric as a float.

parlai.core.metrics.normalize_answer(s)

Lower text and remove punctuation, articles and extra whitespace.

parlai.core.metrics.aggregate_named_reports(named_reports: Dict[str, Dict[str, parlai.core.metrics.Metric]], micro_average: bool = False) → Dict[str, parlai.core.metrics.Metric]

Aggregate metrics from multiple reports.

Parameters

• reports – Dict of tasks -> metrics.

• micro_average – If true, top level metrics will be the micro average. By default, we use macro average.

Returns

The aggregated report

parlai.core.metrics.aggregate_unnamed_reports(reports: List[Dict[str, parlai.core.metrics.Metric]]) → Dict[str, parlai.core.metrics.Metric]

Combines metrics without regard for tracking provenence.

class parlai.core.metrics.Metrics(threadsafe=False, shared=None)

Bases: object

Metrics aggregator.

__init__(threadsafe=False, shared=None)

Initialize self. See help(type(self)) for accurate signature.

add(key: str, value: Optional[parlai.core.metrics.Metric]) → None

Record an accumulation to a metric.

report()

Report the metrics over all data seen so far.

clear_recent()

Clear recent metrics (latest example).

report_recent()

Report recent metrics (latest example).

clear()

Clear all the metrics.

add_metrics(other: Metrics) → None

Aggregate another Metrics objects metrics into this one.

Note that it is assumed that the keys for metrics are disjoint between Metrics objects.

class parlai.core.metrics.TeacherMetrics(metrics_list: str = 'default', shared: Optional[Dict[str, Any]] = None)

Bases: parlai.core.metrics.Metrics

Helper container which encapsulates standard metrics (F1, BLEU, …).

__init__(metrics_list: str = 'default', shared: Optional[Dict[str, Any]] = None) → None

Initialize self. See help(type(self)) for accurate signature.

evaluate_response(observation: parlai.core.message.Message, labels: List[str]) → None

Compute all required text-based metrics based on an observation and labels.