White Box Metrics

When the true saliency is available, it is possible to use white box metrics to compare an attribution method output against it. These metrics are listed below:

Summary

tint.metrics.white_box.aup(attributions, ...)	Area under precision.
tint.metrics.white_box.auprc(attributions, ...)	Area under precision-recall.
tint.metrics.white_box.aur(attributions, ...)	Area under recall.
tint.metrics.white_box.entropy(attributions, ...)	Entropy measure of the attributions over the true_attributions.
tint.metrics.white_box.information(...[, ...])	Information measure of the attributions over the true_attributions.
tint.metrics.white_box.mae(attributions, ...)	Mean absolute error.
tint.metrics.white_box.mse(attributions, ...)	Mean squared error.
tint.metrics.white_box.rmse(attributions, ...)	Root mean squared error.
tint.metrics.white_box.roc_auc(attributions, ...)	Roc auc score.

Detailed classes and methods

tint.metrics.white_box.aup(attributions: TensorOrTupleOfTensorsGeneric, true_attributions: TensorOrTupleOfTensorsGeneric, normalize: bool = True) → Tuple[float]

Area under precision.

This is the standard area under the precision curve. Higher is better.

Parameters:

• attributions (tensor or tuple of tensors) – The attributions with respect to each input feature. Attributions will always be the same size as the provided inputs, with each value providing the attribution of the corresponding input index. If a single tensor is provided as inputs, a single float is returned. If a tuple is provided for inputs, a tuple of float is returned.

• true_attributions (tensor or tuple of tensors) – True attributions to be used as a benchmark. Should be of the same format as the attributions.

• normalize (bool) – Whether to normalize the attributions before computing the metric or not. Default: True

Returns:

The aup metric.

Return type:

(float or tuple or floats)

Examples

>>> import torch as th
>>> from tint.metrics.white_box import aup

>>> attr = th.rand(8, 7, 5)
>>> true_attr = th.randint(2, (8, 7, 5))

>>> aup_ = aup(attr, true_attr)

tint.metrics.white_box.auprc(attributions: TensorOrTupleOfTensorsGeneric, true_attributions: TensorOrTupleOfTensorsGeneric, normalize: bool = True) → Tuple[float]

Area under precision-recall.

This is the standard area under the precision-recall curve. Higher is better.

Parameters:

• attributions (tensor or tuple of tensors) – The attributions with respect to each input feature. Attributions will always be the same size as the provided inputs, with each value providing the attribution of the corresponding input index. If a single tensor is provided as inputs, a single float is returned. If a tuple is provided for inputs, a tuple of float is returned.

• true_attributions (tensor or tuple of tensors) – True attributions to be used as a benchmark. Should be of the same format as the attributions.

• normalize (bool) – Whether to normalize the attributions before computing the metric or not. Default: True

Returns:

The auprc metric.

Return type:

(float or tuple or floats)

Examples

>>> import torch as th
>>> from tint.metrics.white_box import auprc

>>> attr = th.rand(8, 7, 5)
>>> true_attr = th.randint(2, (8, 7, 5))

>>> auprc_ = auprc(attr, true_attr)

tint.metrics.white_box.aur(attributions: TensorOrTupleOfTensorsGeneric, true_attributions: TensorOrTupleOfTensorsGeneric, normalize: bool = True) → Tuple[float]

Area under recall.

This is the standard area under the recall curve. Higher is better.

Parameters:

• attributions (tensor or tuple of tensors) – The attributions with respect to each input feature. Attributions will always be the same size as the provided inputs, with each value providing the attribution of the corresponding input index. If a single tensor is provided as inputs, a single float is returned. If a tuple is provided for inputs, a tuple of float is returned.

• true_attributions (tensor or tuple of tensors) – True attributions to be used as a benchmark. Should be of the same format as the attributions.

• normalize (bool) – Whether to normalize the attributions before computing the metric or not. Default: True

Returns:

The aur metric.

Return type:

(float or tuple or floats)

Examples

>>> import torch as th
>>> from tint.metrics.white_box import aur

>>> attr = th.rand(8, 7, 5)
>>> true_attr = th.randint(2, (8, 7, 5))

>>> aur_ = aur(attr, true_attr)

tint.metrics.white_box.entropy(attributions: TensorOrTupleOfTensorsGeneric, true_attributions: TensorOrTupleOfTensorsGeneric, normalize: bool = True) → Tuple[float]

Entropy measure of the attributions over the true_attributions.

This metric measures how much information there is in the attributions. A low entropy means that the attributions provide a lot of information. Lower is better.

Parameters:

• attributions (tensor or tuple of tensors) – The attributions with respect to each input feature. Attributions will always be the same size as the provided inputs, with each value providing the attribution of the corresponding input index. If a single tensor is provided as inputs, a single float is returned. If a tuple is provided for inputs, a tuple of float is returned.

• true_attributions (tensor or tuple of tensors) – True attributions to be used as a benchmark. Should be of the same format as the attributions.

• normalize (bool) – Whether to normalize the attributions before computing the metric or not. Default: True

Returns:

The aur metric.

Return type:

(float or tuple or floats)

References

Explaining Time Series Predictions with Dynamic Masks

Examples

>>> import torch as th
>>> from tint.metrics.white_box import entropy

>>> attr = th.rand(8, 7, 5)
>>> true_attr = th.randint(2, (8, 7, 5))

>>> entropy_ = entropy(attr, true_attr)

tint.metrics.white_box.information(attributions: TensorOrTupleOfTensorsGeneric, true_attributions: TensorOrTupleOfTensorsGeneric, normalize: bool = True) → Tuple[float]

Information measure of the attributions over the true_attributions.

This metric measures how much information there is in the attributions. Higher is better.

Parameters:

• attributions (tensor or tuple of tensors) – The attributions with respect to each input feature. Attributions will always be the same size as the provided inputs, with each value providing the attribution of the corresponding input index. If a single tensor is provided as inputs, a single float is returned. If a tuple is provided for inputs, a tuple of float is returned.

• true_attributions (tensor or tuple of tensors) – True attributions to be used as a benchmark. Should be of the same format as the attributions.

• normalize (bool) – Whether to normalize the attributions before computing the metric or not. Default: True

Returns:

The aur metric.

Return type:

(float or tuple or floats)

References

Explaining Time Series Predictions with Dynamic Masks

Examples

>>> import torch as th
>>> from tint.metrics.white_box import information

>>> attr = th.rand(8, 7, 5)
>>> true_attr = th.randint(2, (8, 7, 5))

>>> information_ = information(attr, true_attr)

tint.metrics.white_box.mae(attributions: TensorOrTupleOfTensorsGeneric, true_attributions: TensorOrTupleOfTensorsGeneric, normalize: bool = True) → Tuple[float]

Mean absolute error.

This is the standard mean absolute error. Lower is better.

Parameters:

• attributions (tensor or tuple of tensors) – The attributions with respect to each input feature. Attributions will always be the same size as the provided inputs, with each value providing the attribution of the corresponding input index. If a single tensor is provided as inputs, a single float is returned. If a tuple is provided for inputs, a tuple of float is returned.

• true_attributions (tensor or tuple of tensors) – True attributions to be used as a benchmark. Should be of the same format as the attributions.

• normalize (bool) – Whether to normalize the attributions before computing the metric or not. Default: True

Returns:

The aup metric.

Return type:

(float or tuple or floats)

Examples

>>> import torch as th
>>> from tint.metrics.white_box import aup

>>> attr = th.rand(8, 7, 5)
>>> true_attr = th.randint(2, (8, 7, 5))

>>> aup_ = aup(attr, true_attr)

tint.metrics.white_box.mse(attributions: TensorOrTupleOfTensorsGeneric, true_attributions: TensorOrTupleOfTensorsGeneric, normalize: bool = True) → Tuple[float]

Mean squared error.

This is the standard mean squared error. Lower is better.

Parameters:

• attributions (tensor or tuple of tensors) – The attributions with respect to each input feature. Attributions will always be the same size as the provided inputs, with each value providing the attribution of the corresponding input index. If a single tensor is provided as inputs, a single float is returned. If a tuple is provided for inputs, a tuple of float is returned.

• true_attributions (tensor or tuple of tensors) – True attributions to be used as a benchmark. Should be of the same format as the attributions.

• normalize (bool) – Whether to normalize the attributions before computing the metric or not. Default: True

Returns:

The aup metric.

Return type:

(float or tuple or floats)

Examples

>>> import torch as th
>>> from tint.metrics.white_box import aup

>>> attr = th.rand(8, 7, 5)
>>> true_attr = th.randint(2, (8, 7, 5))

>>> aup_ = aup(attr, true_attr)

tint.metrics.white_box.rmse(attributions: TensorOrTupleOfTensorsGeneric, true_attributions: TensorOrTupleOfTensorsGeneric, normalize: bool = True) → Tuple[float]

Root mean squared error.

This is the standard root mean squared error. Lower is better.

Parameters:

• attributions (tensor or tuple of tensors) – The attributions with respect to each input feature. Attributions will always be the same size as the provided inputs, with each value providing the attribution of the corresponding input index. If a single tensor is provided as inputs, a single float is returned. If a tuple is provided for inputs, a tuple of float is returned.

• true_attributions (tensor or tuple of tensors) – True attributions to be used as a benchmark. Should be of the same format as the attributions.

• normalize (bool) – Whether to normalize the attributions before computing the metric or not. Default: True

Returns:

The aup metric.

Return type:

(float or tuple or floats)

Examples

>>> import torch as th
>>> from tint.metrics.white_box import aup

>>> attr = th.rand(8, 7, 5)
>>> true_attr = th.randint(2, (8, 7, 5))

>>> aup_ = aup(attr, true_attr)

tint.metrics.white_box.roc_auc(attributions: TensorOrTupleOfTensorsGeneric, true_attributions: TensorOrTupleOfTensorsGeneric, normalize: bool = True) → Tuple[float]

Roc auc score.

This is the standard roc auc score. Higher is better.

Parameters:

• attributions (tensor or tuple of tensors) – The attributions with respect to each input feature. Attributions will always be the same size as the provided inputs, with each value providing the attribution of the corresponding input index. If a single tensor is provided as inputs, a single float is returned. If a tuple is provided for inputs, a tuple of float is returned.

• true_attributions (tensor or tuple of tensors) – True attributions to be used as a benchmark. Should be of the same format as the attributions.

• normalize (bool) – Whether to normalize the attributions before computing the metric or not. Default: True

Returns:

The roc-auc metric.

Return type:

(float or tuple or floats)

Examples

>>> import torch as th
>>> from tint.metrics.white_box import roc_auc

>>> attr = th.rand(8, 7, 5)
>>> true_attr = th.randint(2, (8, 7, 5))

>>> roc_auc_ = roc_auc(attr, true_attr)