Attribution Methods

time_interpret expands on Captum by providing time series specific attribution methods. These methods are listed below:

Summary

tint.attr.AugmentedOcclusion(forward_func, data)

Augmented Occlusion by sampling the baseline from a bootstrapped distribution.

tint.attr.BayesKernelShap(forward_func[, ...])

Bayesian version of KernelShap.

tint.attr.BayesLime(forward_func[, ...])

Bayesian version of Lime.

tint.attr.DiscretetizedIntegratedGradients(...)

Discretetized Integrated Gradients.

tint.attr.DynaMask(forward_func)

Dynamic masks.

tint.attr.ExtremalMask(forward_func)

Extremal masks.

tint.attr.FeatureAblation(forward_func)

Feature ablation.

tint.attr.Fit(forward_func[, generator, ...])

Feature Importance in Time.

tint.attr.GeodesicIntegratedGradients(...[, ...])

Geodesic Integrated Gradients.

tint.attr.LofKernelShap(forward_func, embeddings)

Local Outlier Factor Kernel Shap.

tint.attr.LofLime(forward_func, embeddings)

Local Outlier Factor Lime.

tint.attr.NonLinearitiesTunnel(...)

Replace non linearities (or any module) with others before running an attribution method.

tint.attr.Occlusion(forward_func)

A perturbation based approach to compute attribution, involving replacing each contiguous rectangular region with a given baseline / reference, and computing the difference in output.

tint.attr.Retain([forward_func, retain, ...])

Retain explainer method.

tint.attr.SequentialIntegratedGradients(...)

Sequential Integrated Gradients.

tint.attr.TemporalAugmentedOcclusion(...[, ...])

Temporal Augmented Occlusion.

tint.attr.TemporalIntegratedGradients(...[, ...])

Temporal Integrated Gradients.

tint.attr.TemporalOcclusion(forward_func)

Temporal Occlusion.

tint.attr.TimeForwardTunnel(attribution_method)

Time Forward Tunnel.

Detailed classes and methods

class tint.attr.AugmentedOcclusion(forward_func: Callable, data: TensorOrTupleOfTensorsGeneric, n_sampling: int = 1, is_temporal: bool = False)[source]

Augmented Occlusion by sampling the baseline from a bootstrapped distribution.

Instead of replacing occulted data by zero, this method samples data from a distribution, which replace occulted data. The resulted occulted data should be closer to the actual data as a result, limiting the amount of out of distribution samples.

Parameters:

  • forward_func (callable) – The forward function of the model or any modification of it.

  • data (tuple, Tensor) – The data from which the baselines are sampled. The shape of the data must be the same as the inputs, except on the first dimension.

  • n_sampling (int) – Number of sampling to run for each occlusion. Default: 1

  • is_temporal (bool) – Whether the data is temporal or not. If True, the data will be ablated to the inputs on the temporal dimension (dimension 1). Default: False

References

What went wrong and when? Instance-wise Feature Importance for Time-series Models

Examples

>>> import torch as th
>>> from tint.attr import AugmentedOcclusion
>>> from tint.models import MLP

>>> inputs = th.rand(8, 7, 5)
>>> data = th.rand(32, 7, 5)
>>> mlp = MLP([5, 3, 1])

>>> explainer = AugmentedOcclusion(mlp, data)
>>> attr = explainer.attribute(inputs, (1, 1))
attribute(inputs: TensorOrTupleOfTensorsGeneric, sliding_window_shapes: Union[Tuple[int, ...], Tuple[Tuple[int, ...], ...]], strides: Union[None, int, Tuple[int, ...], Tuple[Union[int, Tuple[int, ...]], ...]] = None, target: Union[None, int, Tuple[int, ...], Tensor, List[Tuple[int, ...]], List[int]] = None, additional_forward_args: Any = None, perturbations_per_eval: int = 1, attributions_fn: Callable = None, show_progress: bool = False) → TensorOrTupleOfTensorsGeneric[source]
Parameters:

  • inputs (tensor or tuple of tensors) – Input for which occlusion attributions are computed. If forward_func takes a single tensor as input, a single input tensor should be provided. If forward_func takes multiple tensors as input, a tuple of the input tensors should be provided. It is assumed that for all given input tensors, dimension 0 corresponds to the number of examples (aka batch size), and if multiple input tensors are provided, the examples must be aligned appropriately.

  • sliding_window_shapes (tuple or tuple of tuples) – Shape of patch (hyperrectangle) to occlude each input. For a single input tensor, this must be a tuple of length equal to the number of dimensions of the input tensor - 1, defining the dimensions of the patch. If the input tensor is 1-d, this should be an empty tuple. For multiple input tensors, this must be a tuple containing one tuple for each input tensor defining the dimensions of the patch for that input tensor, as described for the single tensor case.

  • strides (int or tuple or tuple of ints or tuple of tuples, optional) – This defines the step by which the occlusion hyperrectangle should be shifted by in each direction for each iteration. For a single tensor input, this can be either a single integer, which is used as the step size in each direction, or a tuple of integers matching the number of dimensions in the occlusion shape, defining the step size in the corresponding dimension. For multiple tensor inputs, this can be either a tuple of integers, one for each input tensor (used for all dimensions of the corresponding tensor), or a tuple of tuples, providing the stride per dimension for each tensor. To ensure that all inputs are covered by at least one sliding window, the stride for any dimension must be <= the corresponding sliding window dimension if the sliding window dimension is less than the input dimension. If None is provided, a stride of 1 is used for each dimension of each input tensor. Default: None

  • target (int, tuple, tensor or list, optional) –

    Output indices for which difference is computed (for classification cases, this is usually the target class). If the network returns a scalar value per example, no target index is necessary. For general 2D outputs, targets can be either:

    • a single integer or a tensor containing a single integer, which is applied to all input examples

    • a list of integers or a 1D tensor, with length matching the number of examples in inputs (dim 0). Each integer is applied as the target for the corresponding example.

    For outputs with > 2 dimensions, targets can be either:

    • A single tuple, which contains #output_dims - 1 elements. This target index is applied to all examples.

    • A list of tuples with length equal to the number of examples in inputs (dim 0), and each tuple containing #output_dims - 1 elements. Each tuple is applied as the target for the corresponding example.

    Default: None

  • additional_forward_args (any, optional) – If the forward function requires additional arguments other than the inputs for which attributions should not be computed, this argument can be provided. It must be either a single additional argument of a Tensor or arbitrary (non-tuple) type or a tuple containing multiple additional arguments including tensors or any arbitrary python types. These arguments are provided to forward_func in order following the arguments in inputs. For a tensor, the first dimension of the tensor must correspond to the number of examples. For all other types, the given argument is used for all forward evaluations. Note that attributions are not computed with respect to these arguments. Default: None

  • perturbations_per_eval (int, optional) – Allows multiple occlusions to be included in one batch (one call to forward_fn). By default, perturbations_per_eval is 1, so each occlusion is processed individually. Each forward pass will contain a maximum of perturbations_per_eval * #examples samples. For DataParallel models, each batch is split among the available devices, so evaluations on each available device contain at most (perturbations_per_eval * #examples) / num_devices samples. Default: 1

  • attributions_fn (Callable, optional) – Applies a function to the attributions before performing the weighted sum. Default: None

  • show_progress (bool, optional) – Displays the progress of computation. It will try to use tqdm if available for advanced features (e.g. time estimation). Otherwise, it will fallback to a simple output of progress. Default: False

Returns:

  • attributions (tensor or tuple of tensors):

    The attributions with respect to each input feature. Attributions will always be the same size as the provided inputs, with each value providing the attribution of the corresponding input index. If a single tensor is provided as inputs, a single tensor is returned. If a tuple is provided for inputs, a tuple of corresponding sized tensors is returned.

Return type:

tensor or tuple of tensors of attributions

class tint.attr.BayesKernelShap(forward_func: Callable, interpretable_model: Optional[Model] = None, percent: int = 95)[source]

Bayesian version of KernelShap.

This method replace the linear regression of the original KernelShap with a bayesian linear regression, allowing to model uncertainty in explainability.

Parameters:

  • forward_func (callable) – The forward function of the model or any modification of it.

  • interpretable_model (Model) –

    Model object to train interpretable model.

    This argument is optional and defaults to SkLearnBayesianRidge(), which is a wrapper around the Bayesian Ridge in SkLearn. This requires having sklearn version >= 0.23 available.

    Other predefined interpretable linear models are provided in tint.attr.models.bayes_linear.

    Alternatively, a custom model object must provide a fit method to train the model, given a dataloader, with batches containing three tensors:

    • interpretable_inputs: Tensor [2D num_samples x num_interp_features],

    • expected_outputs: Tensor [1D num_samples],

    • weights: Tensor [1D num_samples]

    The model object must also provide a representation method to access the appropriate coefficients or representation of the interpretable model after fitting.

    Note that calling fit multiple times should retrain the interpretable model, each attribution call reuses the same given interpretable model object.

    Default: None

  • percent (int) –

    Percentage for the credible intervals. Must be between 0 and 100. Only used when no custom interpretable model is passed. Otherwise, you must specify the percentage for credible interval in the model definition:

    >>> from tint.attr.models import BLRRegression
>>> model = BLRRegression(percent=90)
>>> explainer = BayesKernelShap(mlp, interpretable_model=model)

    Default: 95

References

Reliable Post hoc Explanations: Modeling Uncertainty in Explainability

Examples

>>> import torch as th
>>> from tint.attr import BayesKernelShap
>>> from tint.models import MLP

>>> inputs = th.rand(8, 5)
>>> mlp = MLP([5, 3, 1])

>>> explainer = BayesKernelShap(mlp)
>>> attr, credible_int = explainer.attribute(inputs)
attribute(inputs: TensorOrTupleOfTensorsGeneric, baselines: Union[None, Tensor, int, float, Tuple[Union[Tensor, int, float], ...]] = None, target: Union[None, int, Tuple[int, ...], Tensor, List[Tuple[int, ...]], List[int]] = None, additional_forward_args: Any = None, feature_mask: Union[None, Tensor, Tuple[Tensor, ...]] = None, n_samples: int = 25, perturbations_per_eval: int = 1, return_input_shape: bool = True, show_progress: bool = False) → TensorOrTupleOfTensorsGeneric[source]

This method attributes the output of the model with given target index (in case it is provided, otherwise it assumes that output is a scalar) to the inputs of the model using the approach described above, training an interpretable model based on KernelSHAP and returning a representation of the interpretable model.

It is recommended to only provide a single example as input (tensors with first dimension or batch size = 1). This is because LIME / KernelShap is generally used for sample-based interpretability, training a separate interpretable model to explain a model’s prediction on each individual example.

A batch of inputs can also be provided as inputs, similar to other perturbation-based attribution methods. In this case, if forward_fn returns a scalar per example, attributions will be computed for each example independently, with a separate interpretable model trained for each example. Note that provided similarity and perturbation functions will be provided each example separately (first dimension = 1) in this case. If forward_fn returns a scalar per batch (e.g. loss), attributions will still be computed using a single interpretable model for the full batch. In this case, similarity and perturbation functions will be provided the same original input containing the full batch.

The number of interpretable features is determined from the provided feature mask, or if none is provided, from the default feature mask, which considers each scalar input as a separate feature. It is generally recommended to provide a feature mask which groups features into a small number of interpretable features / components (e.g. superpixels in images).

Parameters:

  • inputs (Tensor or tuple[Tensor, ...]) – Input for which KernelShap is computed. If forward_func takes a single tensor as input, a single input tensor should be provided. If forward_func takes multiple tensors as input, a tuple of the input tensors should be provided. It is assumed that for all given input tensors, dimension 0 corresponds to the number of examples, and if multiple input tensors are provided, the examples must be aligned appropriately.

  • baselines (scalar, Tensor, tuple of scalar, or Tensor, optional) –

    Baselines define the reference value which replaces each feature when the corresponding interpretable feature is set to 0. Baselines can be provided as:

    • a single tensor, if inputs is a single tensor, with exactly the same dimensions as inputs or the first dimension is one and the remaining dimensions match with inputs.

    • a single scalar, if inputs is a single tensor, which will be broadcasted for each input value in input tensor.

    • a tuple of tensors or scalars, the baseline corresponding to each tensor in the inputs’ tuple can be:

      • either a tensor with matching dimensions to corresponding tensor in the inputs’ tuple or the first dimension is one and the remaining dimensions match with the corresponding input tensor.

      • or a scalar, corresponding to a tensor in the inputs’ tuple. This scalar value is broadcasted for corresponding input tensor.

    In the cases when baselines is not provided, we internally use zero scalar corresponding to each input tensor. Default: None

  • target (int, tuple, Tensor, or list, optional) –

    Output indices for which surrogate model is trained (for classification cases, this is usually the target class). If the network returns a scalar value per example, no target index is necessary. For general 2D outputs, targets can be either:

    • a single integer or a tensor containing a single integer, which is applied to all input examples

    • a list of integers or a 1D tensor, with length matching the number of examples in inputs (dim 0). Each integer is applied as the target for the corresponding example.

    For outputs with > 2 dimensions, targets can be either:

    • A single tuple, which contains #output_dims - 1 elements. This target index is applied to all examples.

    • A list of tuples with length equal to the number of examples in inputs (dim 0), and each tuple containing #output_dims - 1 elements. Each tuple is applied as the target for the corresponding example.

    Default: None

  • additional_forward_args (Any, optional) – If the forward function requires additional arguments other than the inputs for which attributions should not be computed, this argument can be provided. It must be either a single additional argument of a Tensor or arbitrary (non-tuple) type or a tuple containing multiple additional arguments including tensors or any arbitrary python types. These arguments are provided to forward_func in order following the arguments in inputs. For a tensor, the first dimension of the tensor must correspond to the number of examples. It will be repeated for each of n_steps along the integrated path. For all other types, the given argument is used for all forward evaluations. Note that attributions are not computed with respect to these arguments. Default: None

  • feature_mask (Tensor or tuple[Tensor, ...], optional) – feature_mask defines a mask for the input, grouping features which correspond to the same interpretable feature. feature_mask should contain the same number of tensors as inputs. Each tensor should be the same size as the corresponding input or broadcastable to match the input tensor. Values across all tensors should be integers in the range 0 to num_interp_features - 1, and indices corresponding to the same feature should have the same value. Note that features are grouped across tensors (unlike feature ablation and occlusion), so if the same index is used in different tensors, those features are still grouped and added simultaneously. If None, then a feature mask is constructed which assigns each scalar within a tensor as a separate feature. Default: None

  • n_samples (int, optional) – The number of samples of the original model used to train the surrogate interpretable model. Default: 50 if n_samples is not provided.

  • perturbations_per_eval (int, optional) – Allows multiple samples to be processed simultaneously in one call to forward_fn. Each forward pass will contain a maximum of perturbations_per_eval * #examples samples. For DataParallel models, each batch is split among the available devices, so evaluations on each available device contain at most (perturbations_per_eval * #examples) / num_devices samples. If the forward function returns a single scalar per batch, perturbations_per_eval must be set to 1. Default: 1

  • return_input_shape (bool, optional) – Determines whether the returned tensor(s) only contain the coefficients for each interp- retable feature from the trained surrogate model, or whether the returned attributions match the input shape. When return_input_shape is True, the return type of attribute matches the input shape, with each element containing the coefficient of the corresponding interpretable feature. All elements with the same value in the feature mask will contain the same coefficient in the returned attributions. If return_input_shape is False, a 1D tensor is returned, containing only the coefficients of the trained interpretable model, with length num_interp_features.

  • show_progress (bool, optional) – Displays the progress of computation. It will try to use tqdm if available for advanced features (e.g. time estimation). Otherwise, it will fallback to a simple output of progress. Default: False

Returns:

  • attributions (tensor or tuple of tensors):

    The attributions with respect to each input feature. If return_input_shape = True, attributions will be the same size as the provided inputs, with each value providing the coefficient of the corresponding interpretale feature. If return_input_shape is False, a 1D tensor is returned, containing only the coefficients of the trained interpreatable models, with length num_interp_features.

  • credible_intervals (tensor or tuple of tensors):

    The credible intervals associated with each attribution. If return_input_shape = True, credible intervals will be the same size as the provided inputs, with each value providing the coefficient of the corresponding interpretale feature. If return_input_shape is False, a 1D tensor is returned, containing only the credible intervals of the trained interpreatable models, with length num_interp_features.

Return type:

2-element tuple of attributions, credible_intervals

Examples::
>>> # SimpleClassifier takes a single input tensor of size Nx4x4,
>>> # and returns an Nx3 tensor of class probabilities.
>>> net = SimpleClassifier()

>>> # Generating random input with size 1 x 4 x 4
>>> input = torch.randn(1, 4, 4)

>>> # Defining KernelShap interpreter
>>> ks = KernelShap(net)
>>> # Computes attribution, with each of the 4 x 4 = 16
>>> # features as a separate interpretable feature
>>> attr = ks.attribute(input, target=1, n_samples=200)

>>> # Alternatively, we can group each 2x2 square of the inputs
>>> # as one 'interpretable' feature and perturb them together.
>>> # This can be done by creating a feature mask as follows, which
>>> # defines the feature groups, e.g.:
>>> # +---+---+---+---+
>>> # | 0 | 0 | 1 | 1 |
>>> # +---+---+---+---+
>>> # | 0 | 0 | 1 | 1 |
>>> # +---+---+---+---+
>>> # | 2 | 2 | 3 | 3 |
>>> # +---+---+---+---+
>>> # | 2 | 2 | 3 | 3 |
>>> # +---+---+---+---+
>>> # With this mask, all inputs with the same value are set to their
>>> # baseline value, when the corresponding binary interpretable
>>> # feature is set to 0.
>>> # The attributions can be calculated as follows:
>>> # feature mask has dimensions 1 x 4 x 4
>>> feature_mask = torch.tensor([[[0,0,1,1],[0,0,1,1],
>>>                             [2,2,3,3],[2,2,3,3]]])

>>> # Computes KernelSHAP attributions with feature mask.
>>> attr = ks.attribute(input, target=1, feature_mask=feature_mask)
class tint.attr.BayesLime(forward_func: Callable, interpretable_model: Optional[Model] = None, similarity_func: Optional[Callable] = None, perturb_func: Optional[Callable] = None, percent: int = 95)[source]

Bayesian version of Lime.

This method replace the linear regression of the original Lime with a bayesian linear regression, allowing to model uncertainty in explainability.

Parameters:

  • forward_func (callable) – The forward function of the model or any modification of it.

  • interpretable_model (Model) –

    Model object to train interpretable model.

    This argument is optional and defaults to SkLearnBayesianRidge(), which is a wrapper around the Bayesian Ridge in SkLearn. This requires having sklearn version >= 0.23 available.

    Other predefined interpretable linear models are provided in tint.attr.models.bayes_linear.

    Alternatively, a custom model object must provide a fit method to train the model, given a dataloader, with batches containing three tensors:

    • interpretable_inputs: Tensor [2D num_samples x num_interp_features],

    • expected_outputs: Tensor [1D num_samples],

    • weights: Tensor [1D num_samples]

    The model object must also provide a representation method to access the appropriate coefficients or representation of the interpretable model after fitting.

    Note that calling fit multiple times should retrain the interpretable model, each attribution call reuses the same given interpretable model object.

    Default: None

  • similarity_func (Callable, optional) –

    Function which takes a single sample along with its corresponding interpretable representation and returns the weight of the interpretable sample for training the interpretable model. This is often referred to as a similarity kernel.

    This argument is optional and defaults to a function which applies an exponential kernel to the cosine distance between the original input and perturbed input, with a kernel width of 1.0.

    A similarity function applying an exponential kernel to cosine / euclidean distances can be constructed using the provided get_exp_kernel_similarity_function in captum.attr._core.lime.

    Alternately, a custom callable can also be provided. The expected signature of this callable is:

    >>> def similarity_func(
>>>    original_input: Tensor or tuple[Tensor, ...],
>>>    perturbed_input: Tensor or tuple[Tensor, ...],
>>>    perturbed_interpretable_input:
>>>        Tensor [2D 1 x num_interp_features],
>>>    **kwargs: Any
>>> ) -> float or Tensor containing float scalar

    perturbed_input and original_input will be the same type and contain tensors of the same shape, with original_input being the same as the input provided when calling attribute.

    kwargs includes baselines, feature_mask, num_interp_features (integer, determined from feature mask).

  • perturb_func (Callable, optional) –

    Function which returns a single sampled input, which is a binary vector of length num_interp_features, or a generator of such tensors.

    This function is optional, the default function returns a binary vector where each element is selected independently and uniformly at random. Custom logic for selecting sampled binary vectors can be implemented by providing a function with the following expected signature:

    >>> perturb_func(
>>>    original_input: Tensor or tuple[Tensor, ...],
>>>    **kwargs: Any
>>> ) -> Tensor [Binary 2D Tensor 1 x num_interp_features]
>>>  or generator yielding such tensors

    kwargs includes baselines, feature_mask, num_interp_features (integer, determined from feature mask).

  • percent (int) –

    Percentage for the credible intervals. Must be between 0 and 100. Only used when no custom interpretable model is passed. Otherwise, you must specify the percentage for credible interval in the model definition:

    >>> from tint.attr.models import BLRRegression
>>> model = BLRRegression(percent=90)
>>> explainer = BayesLime(mlp, interpretable_model=model)

    Default: 95

References

Reliable Post hoc Explanations: Modeling Uncertainty in Explainability

Examples

>>> import torch as th
>>> from tint.attr import BayesLime
>>> from tint.models import MLP

>>> inputs = th.rand(8, 5)
>>> mlp = MLP([5, 3, 1])

>>> explainer = BayesLime(mlp)
>>> attr, credible_int = explainer.attribute(inputs)
attribute(inputs: TensorOrTupleOfTensorsGeneric, baselines: Union[None, Tensor, int, float, Tuple[Union[Tensor, int, float], ...]] = None, target: Union[None, int, Tuple[int, ...], Tensor, List[Tuple[int, ...]], List[int]] = None, additional_forward_args: Any = None, feature_mask: Union[None, Tensor, Tuple[Tensor, ...]] = None, n_samples: int = 25, perturbations_per_eval: int = 1, return_input_shape: bool = True, show_progress: bool = False) → TensorOrTupleOfTensorsGeneric[source]

This method attributes the output of the model with given target index (in case it is provided, otherwise it assumes that output is a scalar) to the inputs of the model using the approach described above, training an interpretable model and returning a representation of the interpretable model.

It is recommended to only provide a single example as input (tensors with first dimension or batch size = 1). This is because LIME is generally used for sample-based interpretability, training a separate interpretable model to explain a model’s prediction on each individual example.

A batch of inputs can also be provided as inputs, similar to other perturbation-based attribution methods. In this case, if forward_fn returns a scalar per example, attributions will be computed for each example independently, with a separate interpretable model trained for each example. Note that provided similarity and perturbation functions will be provided each example separately (first dimension = 1) in this case. If forward_fn returns a scalar per batch (e.g. loss), attributions will still be computed using a single interpretable model for the full batch. In this case, similarity and perturbation functions will be provided the same original input containing the full batch.

The number of interpretable features is determined from the provided feature mask, or if none is provided, from the default feature mask, which considers each scalar input as a separate feature. It is generally recommended to provide a feature mask which groups features into a small number of interpretable features / components (e.g. superpixels in images).

Parameters:

  • inputs (Tensor or tuple[Tensor, ...]) – Input for which LIME is computed. If forward_func takes a single tensor as input, a single input tensor should be provided. If forward_func takes multiple tensors as input, a tuple of the input tensors should be provided. It is assumed that for all given input tensors, dimension 0 corresponds to the number of examples, and if multiple input tensors are provided, the examples must be aligned appropriately.

  • baselines (scalar, Tensor, tuple of scalar, or Tensor, optional) –

    Baselines define reference value which replaces each feature when the corresponding interpretable feature is set to 0. Baselines can be provided as:

    • a single tensor, if inputs is a single tensor, with exactly the same dimensions as inputs or the first dimension is one and the remaining dimensions match with inputs.

    • a single scalar, if inputs is a single tensor, which will be broadcasted for each input value in input tensor.

    • a tuple of tensors or scalars, the baseline corresponding to each tensor in the inputs’ tuple can be:

      • either a tensor with matching dimensions to corresponding tensor in the inputs’ tuple or the first dimension is one and the remaining dimensions match with the corresponding input tensor.

      • or a scalar, corresponding to a tensor in the inputs’ tuple. This scalar value is broadcasted for corresponding input tensor.

    In the cases when baselines is not provided, we internally use zero scalar corresponding to each input tensor. Default: None

  • target (int, tuple, Tensor, or list, optional) –

    Output indices for which surrogate model is trained (for classification cases, this is usually the target class). If the network returns a scalar value per example, no target index is necessary. For general 2D outputs, targets can be either:

    • a single integer or a tensor containing a single integer, which is applied to all input examples

    • a list of integers or a 1D tensor, with length matching the number of examples in inputs (dim 0). Each integer is applied as the target for the corresponding example.

    For outputs with > 2 dimensions, targets can be either:

    • A single tuple, which contains #output_dims - 1 elements. This target index is applied to all examples.

    • A list of tuples with length equal to the number of examples in inputs (dim 0), and each tuple containing #output_dims - 1 elements. Each tuple is applied as the target for the corresponding example.

    Default: None

  • additional_forward_args (Any, optional) – If the forward function requires additional arguments other than the inputs for which attributions should not be computed, this argument can be provided. It must be either a single additional argument of a Tensor or arbitrary (non-tuple) type or a tuple containing multiple additional arguments including tensors or any arbitrary python types. These arguments are provided to forward_func in order following the arguments in inputs. For a tensor, the first dimension of the tensor must correspond to the number of examples. It will be repeated for each of n_steps along the integrated path. For all other types, the given argument is used for all forward evaluations. Note that attributions are not computed with respect to these arguments. Default: None

  • feature_mask (Tensor or tuple[Tensor, ...], optional) – feature_mask defines a mask for the input, grouping features which correspond to the same interpretable feature. feature_mask should contain the same number of tensors as inputs. Each tensor should be the same size as the corresponding input or broadcastable to match the input tensor. Values across all tensors should be integers in the range 0 to num_interp_features - 1, and indices corresponding to the same feature should have the same value. Note that features are grouped across tensors (unlike feature ablation and occlusion), so if the same index is used in different tensors, those features are still grouped and added simultaneously. If None, then a feature mask is constructed which assigns each scalar within a tensor as a separate feature. Default: None

  • n_samples (int, optional) – The number of samples of the original model used to train the surrogate interpretable model. Default: 50 if n_samples is not provided.

  • perturbations_per_eval (int, optional) – Allows multiple samples to be processed simultaneously in one call to forward_fn. Each forward pass will contain a maximum of perturbations_per_eval * #examples samples. For DataParallel models, each batch is split among the available devices, so evaluations on each available device contain at most (perturbations_per_eval * #examples) / num_devices samples. If the forward function returns a single scalar per batch, perturbations_per_eval must be set to 1. Default: 1

  • return_input_shape (bool, optional) – Determines whether the returned tensor(s) only contain the coefficients for each interp- retable feature from the trained surrogate model, or whether the returned attributions match the input shape. When return_input_shape is True, the return type of attribute matches the input shape, with each element containing the coefficient of the corresponding interpretale feature. All elements with the same value in the feature mask will contain the same coefficient in the returned attributions. If return_input_shape is False, a 1D tensor is returned, containing only the coefficients of the trained interpreatable models, with length num_interp_features.

  • show_progress (bool, optional) – Displays the progress of computation. It will try to use tqdm if available for advanced features (e.g. time estimation). Otherwise, it will fallback to a simple output of progress. Default: False

Returns:

  • attributions (tensor or tuple of tensors):

    The attributions with respect to each input feature. If return_input_shape = True, attributions will be the same size as the provided inputs, with each value providing the coefficient of the corresponding interpretale feature. If return_input_shape is False, a 1D tensor is returned, containing only the coefficients of the trained interpreatable models, with length num_interp_features.

  • credible_intervals (tensor or tuple of tensors):

    The credible intervals associated with each attribution. If return_input_shape = True, credible intervals will be the same size as the provided inputs, with each value providing the coefficient of the corresponding interpretale feature. If return_input_shape is False, a 1D tensor is returned, containing only the credible intervals of the trained interpreatable models, with length num_interp_features.

Return type:

2-element tuple of attributions, credible_intervals

Examples:

>>> # SimpleClassifier takes a single input tensor of size Nx4x4,
>>> # and returns an Nx3 tensor of class probabilities.
>>> net = SimpleClassifier()

>>> # Generating random input with size 1 x 4 x 4
>>> input = torch.randn(1, 4, 4)

>>> # Defining Lime interpreter
>>> lime = Lime(net)
>>> # Computes attribution, with each of the 4 x 4 = 16
>>> # features as a separate interpretable feature
>>> attr = lime.attribute(input, target=1, n_samples=200)

>>> # Alternatively, we can group each 2x2 square of the inputs
>>> # as one 'interpretable' feature and perturb them together.
>>> # This can be done by creating a feature mask as follows, which
>>> # defines the feature groups, e.g.:
>>> # +---+---+---+---+
>>> # | 0 | 0 | 1 | 1 |
>>> # +---+---+---+---+
>>> # | 0 | 0 | 1 | 1 |
>>> # +---+---+---+---+
>>> # | 2 | 2 | 3 | 3 |
>>> # +---+---+---+---+
>>> # | 2 | 2 | 3 | 3 |
>>> # +---+---+---+---+
>>> # With this mask, all inputs with the same value are set to their
>>> # baseline value, when the corresponding binary interpretable
>>> # feature is set to 0.
>>> # The attributions can be calculated as follows:
>>> # feature mask has dimensions 1 x 4 x 4
>>> feature_mask = torch.tensor([[[0,0,1,1],[0,0,1,1],
>>>                             [2,2,3,3],[2,2,3,3]]])

>>> # Computes interpretable model and returning attributions
>>> # matching input shape.
>>> attr = lime.attribute(input, target=1, feature_mask=feature_mask)
class tint.attr.DiscretetizedIntegratedGradients(forward_func: Callable, multiply_by_inputs: bool = True)[source]

Discretetized Integrated Gradients.

This method discretizes the path between an input and a reference baseline. It was developed for text data and language models, to handle the discreteness of the word embedding space.

Parameters:

  • forward_func (callable) – The forward function of the model or any modification of it

  • multiply_by_inputs (bool, optional) –

    Indicates whether to factor model inputs’ multiplier in the final attribution scores. In the literature this is also known as local vs global attribution. If inputs’ multiplier isn’t factored in, then that type of attribution method is also called local attribution. If it is, then that type of attribution method is called global. More detailed can be found here: https://arxiv.org/abs/1711.06104

    In case of integrated gradients, if multiply_by_inputs is set to True, final sensitivity scores are being multiplied by (inputs - baselines).

References

  1. Discretized Integrated Gradients for Explaining Language Models

  2. https://github.com/INK-USC/DIG

Examples

>>> import torch as th
>>> from tint.attr import DiscretetizedIntegratedGradients
>>> from tint.models import MLP

>>> inputs = th.rand(50, 5)
>>> mlp = MLP([5, 3, 1])

>>> explainer = DiscretetizedIntegratedGradients(mlp)
>>> attr = explainer.attribute(inputs)
attribute(scaled_features: TensorOrTupleOfTensorsGeneric, target: Union[None, int, Tuple[int, ...], Tensor, List[Tuple[int, ...]], List[int]] = None, additional_forward_args: Any = None, n_steps: int = 50, return_convergence_delta: bool = False) → Union[TensorOrTupleOfTensorsGeneric, Tuple[TensorOrTupleOfTensorsGeneric, Tensor]][source]

Attribute method.

Parameters:

  • scaled_features – (tensor, tuple): Input for which integrated gradients are computed. If forward_func takes a single tensor as input, a single input tensor should be provided. If forward_func takes multiple tensors as input, a tuple of the input tensors should be provided. It is assumed that for all given input tensors, dimension 0 corresponds to the number of examples, and if multiple input tensors are provided, the examples must be aligned appropriately.

  • target (int, int, tuple, tensor, list) –

    Output indices for which gradients are computed (for classification cases, this is usually the target class). If the network returns a scalar value per example, no target index is necessary. For general 2D outputs, targets can be either:

    • a single integer or a tensor containing a single integer, which is applied to all input examples

    • a list of integers or a 1D tensor, with length matching the number of examples in inputs (dim 0). Each integer is applied as the target for the corresponding example.

    For outputs with > 2 dimensions, targets can be either:

    • A single tuple, which contains #output_dims - 1 elements. This target index is applied to all examples.

    • A list of tuples with length equal to the number of examples in inputs (dim 0), and each tuple containing #output_dims - 1 elements. Each tuple is applied as the target for the corresponding example.

    Default: None

  • additional_forward_args (Any) – If the forward function requires additional arguments other than the inputs for which attributions should not be computed, this argument can be provided. It must be either a single additional argument of a Tensor or arbitrary (non-tuple) type or a tuple containing multiple additional arguments including tensors or any arbitrary python types. These arguments are provided to forward_func in order following the arguments in inputs. For a tensor, the first dimension of the tensor must correspond to the number of examples. It will be repeated for each of n_steps along the integrated path. For all other types, the given argument is used for all forward evaluations. Note that attributions are not computed with respect to these arguments. Default: None

  • n_steps – The number of steps used by the approximation method. Default: 50.

  • return_convergence_delta – Indicates whether to return convergence delta or not. If return_convergence_delta is set to True convergence delta will be returned in a tuple following attributions. Default: False

Returns:

  • attributions (tensor or tuple of tensors):

    Integrated gradients with respect to each input feature. attributions will always be the same size as the provided inputs, with each value providing the attribution of the corresponding input index. If a single tensor is provided as inputs, a single tensor is returned. If a tuple is provided for inputs, a tuple of corresponding sized tensors is returned.

  • delta (tensor, returned if return_convergence_delta=True):

    The difference between the total approximated and true integrated gradients. This is computed using the property that the total sum of forward_func(inputs) - forward_func(baselines) must equal the total sum of the integrated gradient. Delta is calculated per example, meaning that the number of elements in returned delta tensor is equal to the number of of examples in inputs.

Return type:

attributions or 2-element tuple of attributions, delta

class tint.attr.DynaMask(forward_func: Callable)[source]

Dynamic masks.

This method aims to explain time series data, by learning a mask representing features importance. This method was inspired from Fong et al., and can be used in “preservation game” mode: trying to keep the closest predictions, compared with unperturebed data, with the minimal number of features, or in “deletion game” mode, trying to get the furthest predictions by removing the minimal number of features.

This implementation batchify the original method by leanrning in parallel multiple inputs and multiple keep_ratio (called mask_group in the original implementation.

Parameters:

forward_func (callable) – The forward function of the model or any modification of it.

References

  1. Explaining Time Series Predictions with Dynamic Masks

  2. Understanding Deep Networks via Extremal Perturbations and Smooth Masks

Examples

>>> import torch as th
>>> from tint.attr import DynaMask
>>> from tint.models import MLP

>>> inputs = th.rand(8, 7, 5)
>>> data = th.rand(32, 7, 5)
>>> mlp = MLP([5, 3, 1])

>>> explainer = DynaMask(mlp)
>>> attr = explainer.attribute(inputs)
attribute(inputs: TensorOrTupleOfTensorsGeneric, additional_forward_args: Any = None, trainer: Trainer = None, mask_net: MaskNet = None, batch_size: int = 32, temporal_additional_forward_args: Tuple[bool] = None, return_temporal_attributions: bool = False, return_best_ratio: bool = False) → TensorOrTupleOfTensorsGeneric[source]

Attribute method.

Parameters:

  • inputs (tensor or tuple of tensors) – Input for which integrated gradients are computed. If forward_func takes a single tensor as input, a single input tensor should be provided. If forward_func takes multiple tensors as input, a tuple of the input tensors should be provided. It is assumed that for all given input tensors, dimension 0 corresponds to the number of examples, and if multiple input tensors are provided, the examples must be aligned appropriately.

  • additional_forward_args (any, optional) – If the forward function requires additional arguments other than the inputs for which attributions should not be computed, this argument can be provided. It must be either a single additional argument of a Tensor or arbitrary (non-tuple) type or a tuple containing multiple additional arguments including tensors or any arbitrary python types. These arguments are provided to forward_func in order following the arguments in inputs. For a tensor, the first dimension of the tensor must correspond to the number of examples. It will be repeated for each of n_steps along the integrated path. For all other types, the given argument is used for all forward evaluations. Note that attributions are not computed with respect to these arguments. Default: None

  • trainer (Trainer) – Pytorch Lightning trainer. If None, a default trainer will be provided. Default: None

  • mask_net (MaskNet) – A Mask model. If None, a default model will be provided. Default: None

  • batch_size (int) – Batch size for Mask training. Default: 32

  • temporal_additional_forward_args (tuple) – Set each additional forward arg which is temporal. Only used with return_temporal_attributions. Default: None

  • return_temporal_attributions (bool) – Whether to return attributions for all times or not. Default: False

  • return_best_ratio (bool) – Whether to return the best keep_ratio or not. Default: False

Returns:

The attributions with respect to each input feature. Attributions will always be the same size as the provided inputs, with each value providing the attribution of the corresponding input index. If a single tensor is provided as inputs, a single tensor is returned. If a tuple is provided for inputs, a tuple of corresponding sized tensors is returned.

Return type:

  • attributions (tensor or tuple of tensors)

class tint.attr.ExtremalMask(forward_func: Callable)[source]

Extremal masks.

This method extends the work of Fong et al. and Crabbé et al. by allowing the perturbation function to be learnt. This is in addition to the learnt mask. For instance, this perturbation function can be learnt with a RNN while Crabbé et al. only consider fixed perturbations: Gaussian blur and fade to moving average.

Parameters:

forward_func (callable) – The forward function of the model or any modification of it.

References

  1. Learning Perturbations to Explain Time Series Predictions

  2. Understanding Deep Networks via Extremal Perturbations and Smooth Masks

Examples

>>> import torch as th
>>> from tint.attr import ExtremalMask
>>> from tint.models import MLP

>>> inputs = th.rand(8, 7, 5)
>>> data = th.rand(32, 7, 5)
>>> mlp = MLP([5, 3, 1])

>>> explainer = ExtremalMask(mlp)
>>> attr = explainer.attribute(inputs)
attribute(inputs: TensorOrTupleOfTensorsGeneric, baselines: Union[None, Tensor, int, float, Tuple[Union[Tensor, int, float], ...]] = None, target: Union[None, int, Tuple[int, ...], Tensor, List[Tuple[int, ...]], List[int]] = None, additional_forward_args: Any = None, trainer: Trainer = None, mask_net: ExtremalMaskNet = None, batch_size: int = 32, temporal_additional_forward_args: Tuple[bool] = None, return_temporal_attributions: bool = False) → TensorOrTupleOfTensorsGeneric[source]

Attribute method.

Parameters:

  • inputs (tensor or tuple of tensors) – Input for which occlusion attributions are computed. If forward_func takes a single tensor as input, a single input tensor should be provided. If forward_func takes multiple tensors as input, a tuple of the input tensors should be provided. It is assumed that for all given input tensors, dimension 0 corresponds to the number of examples (aka batch size), and if multiple input tensors are provided, the examples must be aligned appropriately.

  • baselines (scalar, tensor, tuple of scalars or tensors, optional) –

    Baselines define reference value which replaces each feature when occluded. Baselines can be provided as:

    • a single tensor, if inputs is a single tensor, with exactly the same dimensions as inputs or broadcastable to match the dimensions of inputs

    • a single scalar, if inputs is a single tensor, which will be broadcasted for each input value in input tensor.

    • a tuple of tensors or scalars, the baseline corresponding to each tensor in the inputs’ tuple can be:

      • either a tensor with matching dimensions to corresponding tensor in the inputs’ tuple or the first dimension is one and the remaining dimensions match with the corresponding input tensor.

      • or a scalar, corresponding to a tensor in the inputs’ tuple. This scalar value is broadcasted for corresponding input tensor.

    In the cases when baselines is not provided, we internally use zero scalar corresponding to each input tensor. Default: None

  • target (int, tuple, tensor or list, optional) –

    Output indices for which difference is computed (for classification cases, this is usually the target class). If the network returns a scalar value per example, no target index is necessary. For general 2D outputs, targets can be either:

    • a single integer or a tensor containing a single integer, which is applied to all input examples

    • a list of integers or a 1D tensor, with length matching the number of examples in inputs (dim 0). Each integer is applied as the target for the corresponding example.

    For outputs with > 2 dimensions, targets can be either:

    • A single tuple, which contains #output_dims - 1 elements. This target index is applied to all examples.

    • A list of tuples with length equal to the number of examples in inputs (dim 0), and each tuple containing #output_dims - 1 elements. Each tuple is applied as the target for the corresponding example.

    Default: None

  • additional_forward_args (any, optional) – If the forward function requires additional arguments other than the inputs for which attributions should not be computed, this argument can be provided. It must be either a single additional argument of a Tensor or arbitrary (non-tuple) type or a tuple containing multiple additional arguments including tensors or any arbitrary python types. These arguments are provided to forward_func in order following the arguments in inputs. For a tensor, the first dimension of the tensor must correspond to the number of examples. For all other types, the given argument is used for all forward evaluations. Note that attributions are not computed with respect to these arguments. Default: None

  • trainer (Trainer) – Pytorch Lightning trainer. If None, a default trainer will be provided. Default: None

  • mask_net (BayesMaskNet) – A Mask model. If None, a default model will be provided. Default: None

  • batch_size (int) – Batch size for Mask training. Default: 32

  • temporal_additional_forward_args (tuple) – Set each additional forward arg which is temporal. Only used with return_temporal_attributions. Default: None

  • return_temporal_attributions (bool) – Whether to return attributions for all times or not. Default: False

Returns:

The attributions with respect to each input feature. Attributions will always be the same size as the provided inputs, with each value providing the attribution of the corresponding input index. If a single tensor is provided as inputs, a single tensor is returned. If a tuple is provided for inputs, a tuple of corresponding sized tensors is returned.

Return type:

  • attributions (tensor or tuple of tensors)

class tint.attr.FeatureAblation(forward_func: Callable)[source]

Feature ablation.

A perturbation based approach to computing attribution, involving replacing each input feature with a given baseline / reference, and computing the difference in output. By default, each scalar value within each input tensor is taken as a feature and replaced independently. Passing a feature mask, allows grouping features to be ablated together. This can be used in cases such as images, where an entire segment or region can be ablated, measuring the importance of the segment (feature group). Each input scalar in the group will be given the same attribution value equal to the change in target as a result of ablating the entire feature group.

The forward function can either return a scalar per example or a tensor of a fixed sized tensor (or scalar value) for the full batch, i.e. the output does not grow as the batch size increase. If the output is fixed we consider this model to be an “aggregation” of the inputs. In the fixed sized output mode we require perturbations_per_eval == 1 and the feature_mask to be either None or for all of them to have 1 as their first dimension (i.e. a feature mask requires to be applied to all inputs).

attribute(inputs: TensorOrTupleOfTensorsGeneric, baselines: Union[None, Tensor, int, float, Tuple[Union[Tensor, int, float], ...]] = None, target: Union[None, int, Tuple[int, ...], Tensor, List[Tuple[int, ...]], List[int]] = None, additional_forward_args: Any = None, feature_mask: Union[None, Tensor, Tuple[Tensor, ...]] = None, perturbations_per_eval: int = 1, attributions_fn: Callable = None, show_progress: bool = False, **kwargs: Any) → TensorOrTupleOfTensorsGeneric[source]
Parameters:

  • inputs (tensor or tuple of tensors) – Input for which ablation attributions are computed. If forward_func takes a single tensor as input, a single input tensor should be provided. If forward_func takes multiple tensors as input, a tuple of the input tensors should be provided. It is assumed that for all given input tensors, dimension 0 corresponds to the number of examples (aka batch size), and if multiple input tensors are provided, the examples must be aligned appropriately.

  • baselines (scalar, tensor, tuple of scalars or tensors, optional) –

    Baselines define reference value which replaces each feature when ablated. Baselines can be provided as:

    • a single tensor, if inputs is a single tensor, with exactly the same dimensions as inputs or broadcastable to match the dimensions of inputs

    • a single scalar, if inputs is a single tensor, which will be broadcasted for each input value in input tensor.

    • a tuple of tensors or scalars, the baseline corresponding to each tensor in the inputs’ tuple can be:

      • either a tensor with matching dimensions to corresponding tensor in the inputs’ tuple or the first dimension is one and the remaining dimensions match with the corresponding input tensor.

      • or a scalar, corresponding to a tensor in the inputs’ tuple. This scalar value is broadcasted for corresponding input tensor.

    In the cases when baselines is not provided, we internally use zero scalar corresponding to each input tensor. Default: None

  • target (int, tuple, tensor or list, optional) –

    Output indices for which gradients are computed (for classification cases, this is usually the target class). If the network returns a scalar value per example, no target index is necessary. For general 2D outputs, targets can be either:

    • a single integer or a tensor containing a single integer, which is applied to all input examples

    • a list of integers or a 1D tensor, with length matching the number of examples in inputs (dim 0). Each integer is applied as the target for the corresponding example.

    For outputs with > 2 dimensions, targets can be either:

    • A single tuple, which contains #output_dims - 1 elements. This target index is applied to all examples.

    • A list of tuples with length equal to the number of examples in inputs (dim 0), and each tuple containing #output_dims - 1 elements. Each tuple is applied as the target for the corresponding example.

    Default: None

  • additional_forward_args (any, optional) – If the forward function requires additional arguments other than the inputs for which attributions should not be computed, this argument can be provided. It must be either a single additional argument of a Tensor or arbitrary (non-tuple) type or a tuple containing multiple additional arguments including tensors or any arbitrary python types. These arguments are provided to forward_func in order following the arguments in inputs. For a tensor, the first dimension of the tensor must correspond to the number of examples. For all other types, the given argument is used for all forward evaluations. Note that attributions are not computed with respect to these arguments. Default: None

  • feature_mask (tensor or tuple of tensors, optional) – feature_mask defines a mask for the input, grouping features which should be ablated together. feature_mask should contain the same number of tensors as inputs. Each tensor should be the same size as the corresponding input or broadcastable to match the input tensor. Each tensor should contain integers in the range 0 to num_features - 1, and indices corresponding to the same feature should have the same value. Note that features within each input tensor are ablated independently (not across tensors). If the forward function returns a single scalar per batch, we enforce that the first dimension of each mask must be 1, since attributions are returned batch-wise rather than per example, so the attributions must correspond to the same features (indices) in each input example. If None, then a feature mask is constructed which assigns each scalar within a tensor as a separate feature, which is ablated independently. Default: None

  • perturbations_per_eval (int, optional) – Allows ablation of multiple features to be processed simultaneously in one call to forward_fn. Each forward pass will contain a maximum of perturbations_per_eval * #examples samples. For DataParallel models, each batch is split among the available devices, so evaluations on each available device contain at most (perturbations_per_eval * #examples) / num_devices samples. If the forward function’s number of outputs does not change as the batch size grows (e.g. if it outputs a scalar value), you must set perturbations_per_eval to 1 and use a single feature mask to describe the features for all examples in the batch. Default: 1

  • attributions_fn (Callable, optional) – Applies a function to the attributions before performing the weighted sum. Default: None

  • show_progress (bool, optional) – Displays the progress of computation. It will try to use tqdm if available for advanced features (e.g. time estimation). Otherwise, it will fallback to a simple output of progress. Default: False

  • **kwargs (Any, optional) – Any additional arguments used by child classes of FeatureAblation (such as Occlusion) to construct ablations. These arguments are ignored when using FeatureAblation directly. Default: None

Returns:

  • attributions (tensor or tuple of tensors):

    The attributions with respect to each input feature. If the forward function returns a scalar value per example, attributions will be the same size as the provided inputs, with each value providing the attribution of the corresponding input index. If the forward function returns a scalar per batch, then attribution tensor(s) will have first dimension 1 and the remaining dimensions will match the input. If a single tensor is provided as inputs, a single tensor is returned. If a tuple of tensors is provided for inputs, a tuple of corresponding sized tensors is returned.

Return type:

tensor or tuple of tensors of attributions

Examples:

>>> # SimpleClassifier takes a single input tensor of size Nx4x4,
>>> # and returns an Nx3 tensor of class probabilities.
>>> net = SimpleClassifier()
>>> # Generating random input with size 2 x 4 x 4
>>> input = torch.randn(2, 4, 4)
>>> # Defining FeatureAblation interpreter
>>> ablator = FeatureAblation(net)
>>> # Computes ablation attribution, ablating each of the 16
>>> # scalar input independently.
>>> attr = ablator.attribute(input, target=1)

>>> # Alternatively, we may want to ablate features in groups, e.g.
>>> # grouping each 2x2 square of the inputs and ablating them together.
>>> # This can be done by creating a feature mask as follows, which
>>> # defines the feature groups, e.g.:
>>> # +---+---+---+---+
>>> # | 0 | 0 | 1 | 1 |
>>> # +---+---+---+---+
>>> # | 0 | 0 | 1 | 1 |
>>> # +---+---+---+---+
>>> # | 2 | 2 | 3 | 3 |
>>> # +---+---+---+---+
>>> # | 2 | 2 | 3 | 3 |
>>> # +---+---+---+---+
>>> # With this mask, all inputs with the same value are ablated
>>> # simultaneously, and the attribution for each input in the same
>>> # group (0, 1, 2, and 3) per example are the same.
>>> # The attributions can be calculated as follows:
>>> # feature mask has dimensions 1 x 4 x 4
>>> feature_mask = torch.tensor([[[0,0,1,1],[0,0,1,1],
>>>                             [2,2,3,3],[2,2,3,3]]])
>>> attr = ablator.attribute(input, target=1, feature_mask=feature_mask)
class tint.attr.Fit(forward_func: Callable, generator: Optional[JointFeatureGeneratorNet] = None, datamodule: Optional[LightningDataModule] = None, features: Optional[Tensor] = None, trainer: Optional[Trainer] = None, batch_size: int = 32)[source]

Feature Importance in Time.

Parameters:

  • forward_func (callable) – The forward function of the model or any modification of it.

  • generator (JointFeatureGeneratorNet) – Conditional generator model to predict future observations as a Pytorch Lightning module. If not provided, a default generator is created. Default to None

  • datamodule (LightningDataModule) – A Pytorch Lightning data module to train the generator. If not provided, you must provide features. Default to None

  • features (th.Tensor) – A tensor of features to train the generator. If not provided, you must provide a datamodule. Default to None

  • trainer (Trainer) – A Pytorch Lightning trainer to train the generator. If not provided, a default trainer is created. Default to None

  • batch_size (int) – Batch size for generator training. Default to 32

References

What went wrong and when? Instance-wise Feature Importance for Time-series Models

Examples

>>> import torch as th
>>> from tint.attr import Fit
>>> from tint.models import MLP

>>> inputs = th.rand(8, 7, 5)
>>> data = th.rand(32, 7, 5)
>>> mlp = MLP([5, 3, 1])

>>> explainer = Fit(mlp, features=data)
>>> attr = explainer.attribute(inputs)
attribute(inputs: TensorOrTupleOfTensorsGeneric, additional_forward_args: Any = None, n_samples: int = 10, distance_metric: str = 'kl', multilabel: bool = False, temporal_additional_forward_args: Tuple[bool] = None, return_temporal_attributions: bool = False, show_progress: bool = False) → TensorOrTupleOfTensorsGeneric[source]

attribute method.

Parameters:

  • inputs (tensor or tuple of tensors) – Input for which integrated gradients are computed. If forward_func takes a single tensor as input, a single input tensor should be provided. If forward_func takes multiple tensors as input, a tuple of the input tensors should be provided. It is assumed that for all given input tensors, dimension 0 corresponds to the number of examples, and if multiple input tensors are provided, the examples must be aligned appropriately.

  • additional_forward_args (any, optional) – If the forward function requires additional arguments other than the inputs for which attributions should not be computed, this argument can be provided. It must be either a single additional argument of a Tensor or arbitrary (non-tuple) type or a tuple containing multiple additional arguments including tensors or any arbitrary python types. These arguments are provided to forward_func in order following the arguments in inputs. For a tensor, the first dimension of the tensor must correspond to the number of examples. It will be repeated for each of n_steps along the integrated path. For all other types, the given argument is used for all forward evaluations. Note that attributions are not computed with respect to these arguments. Default: None

  • n_samples (int) – Number of Monte-Carlo samples. Default: 10

  • distance_metric (str) – Distance metric. Default to ‘kl’

  • multilabel (bool) – Whether the task is single or multi-labeled. Default: False

  • temporal_additional_forward_args (tuple) – Set each additional forward arg which is temporal. Only used with return_temporal_attributions. Default: None

  • return_temporal_attributions (bool) – Whether to return attributions for all times or not. Default: False

  • show_progress (bool, optional) – Displays the progress of computation. It will try to use tqdm if available for advanced features (e.g. time estimation). Otherwise, it will fallback to a simple output of progress. Default: False

Returns:

The attributions with respect to each input feature. Attributions will always be the same size as the provided inputs, with each value providing the attribution of the corresponding input index. If a single tensor is provided as inputs, a single tensor is returned. If a tuple is provided for inputs, a tuple of corresponding sized tensors is returned.

Return type:

  • attributions (tensor or tuple of tensors)

representation(inputs: Tensor, additional_forward_args: Optional[Any] = None, n_samples: int = 10, distance_metric: str = 'kl', multilabel: bool = False, temporal_additional_forward_args: Optional[Tuple[bool]] = None, show_progress: bool = False)[source]

Get representations based on a generator and inputs.

Parameters:

  • inputs (th.Tensor) – Input data.

  • additional_forward_args (Any) – Optional additional args to be passed into the model. Default to None

  • n_samples (int) – Number of Monte-Carlo samples. Default to 10

  • distance_metric (str) – Distance metric. Default to 'kl'

  • multilabel (bool) – Whether the task is single or multi-labeled. Default to False

  • temporal_additional_forward_args (tuple) – Set each additional forward arg which is temporal. Only used with return_temporal_attributions. Default to None

  • show_progress (bool) – Displays the progress of computation. Default to False

Returns:

attributions.

Return type:

th.Tensor

class tint.attr.GeodesicIntegratedGradients(forward_func: Callable, nn: Optional[Union[NearestNeighbors, Tuple[NearestNeighbors, ...]]] = None, data: Optional[TensorOrTupleOfTensorsGeneric] = None, n_neighbors: Optional[Union[int, Tuple[int]]] = None, multiply_by_inputs: bool = True, **kwargs)[source]

Geodesic Integrated Gradients.

This method uses K Nearest Neighbors on the input data to approximate the geodesic path between inputs and reference baselines. The input space is seen here as a Riemannian manifold, whose metric is the inner product of the gradient of the model:

\[<\nabla F(x)^T, \nabla F(x)>\]

Using this path reduces the risk of creating artifacts compared with the original Integrated Gradients (IG). It also supports internal_batch_size for faster compute. The number of steps is also set by default to 5, as less steps are required between two neighbors. With this setting, and the number of neighbors set to 10, the number of gradients to compute is 10 * 5 = 50, the same number as the original IG.

The shortest path is computed using Dijkstra or A* algorithms. This can be computationally expensive for a number of inputs greater than a few thousands.

Parameters:

  • forward_func (callable) – The forward function of the model or any modification of it.

  • nn (NearestNeighbors, tuple) – Nearest neighbors method. If not provided, will be created when calling __init__ or attribute. Default: None

  • data (Tensor, tuple) – Data to fit the knn algorithm. If not provided, the knn will be fitted when calling attribute using the provided inputs data. Default: None

  • n_neighbors (int, tuple) – Number of neighbors to use by default. Can be an integer (same for every inputs) or a tuple. Default: None

  • multiply_by_inputs (bool, optional) –

    Indicates whether to factor model inputs’ multiplier in the final attribution scores. In the literature this is also known as local vs global attribution. If inputs’ multiplier isn’t factored in, then that type of attribution method is also called local attribution. If it is, then that type of attribution method is called global. More detailed can be found here: https://arxiv.org/abs/1711.06104

    In case of integrated gradients, if multiply_by_inputs is set to True, final sensitivity scores are being multiplied by (inputs - baselines).

Examples

>>> import torch as th
>>> from tint.attr import GeodesicIntegratedGradients
>>> from tint.models import MLP

>>> inputs = th.rand(50, 5)
>>> data = th.rand(100, 5)
>>> mlp = MLP([5, 3, 1])

>>> explainer = GeodesicIntegratedGradients(
...     mlp, data=data, n_neighbors=10,
... )
>>> attr = explainer.attribute(inputs)
attribute(inputs: TensorOrTupleOfTensorsGeneric, baselines: BaselineType = None, target: TargetType = None, additional_forward_args: Any = None, n_neighbors: Union[int, Tuple[int]] = None, n_steps: int = 5, n_steiner: int = None, method: str = 'gausslegendre', internal_batch_size: Union[None, int] = None, return_curvature: bool = False, return_convergence_delta: bool = False, distance: str = 'geodesic', show_progress: bool = False, **kwargs) → TensorOrTupleOfTensorsGeneric[source]
attribute(inputs: TensorOrTupleOfTensorsGeneric, baselines: BaselineType = None, target: TargetType = None, additional_forward_args: Any = None, n_neighbors: Union[int, Tuple[int]] = None, n_steps: int = 5, n_steiner: int = None, method: str = 'gausslegendre', internal_batch_size: Union[None, int] = None, *, return_curvature: bool = False, return_convergence_delta: bool, distance: str = 'geodesic', show_progress: bool = False, **kwargs) → Tuple[TensorOrTupleOfTensorsGeneric, Tensor]

This method attributes the output of the model with given target index (in case it is provided, otherwise it assumes that output is a scalar) to the inputs of the model using the approach described above.

In addition to that it also returns, if return_convergence_delta is set to True, integral approximation delta based on the completeness property of integrated gradients.

It also returns the curvature if return_curvature is set to True.

Parameters:

  • inputs (tensor or tuple of tensors) – Input for which integrated gradients are computed. If forward_func takes a single tensor as input, a single input tensor should be provided. If forward_func takes multiple tensors as input, a tuple of the input tensors should be provided. It is assumed that for all given input tensors, dimension 0 corresponds to the number of examples, and if multiple input tensors are provided, the examples must be aligned appropriately.

  • baselines (scalar, tensor, tuple of scalars or tensors, optional) –

    Baselines define the starting point from which integral is computed and can be provided as:

    • a single tensor, if inputs is a single tensor, with exactly the same dimensions as inputs or the first dimension is one and the remaining dimensions match with inputs.

    • a single scalar, if inputs is a single tensor, which will be broadcasted for each input value in input tensor.

    • a tuple of tensors or scalars, the baseline corresponding to each tensor in the inputs’ tuple can be:

      • either a tensor with matching dimensions to corresponding tensor in the inputs’ tuple or the first dimension is one and the remaining dimensions match with the corresponding input tensor.

      • or a scalar, corresponding to a tensor in the inputs’ tuple. This scalar value is broadcasted for corresponding input tensor.

    In the cases when baselines is not provided, we internally use zero scalar corresponding to each input tensor.

    Default: None

  • target (int, tuple, tensor or list, optional) –

    Output indices for which gradients are computed (for classification cases, this is usually the target class). If the network returns a scalar value per example, no target index is necessary. For general 2D outputs, targets can be either:

    • a single integer or a tensor containing a single integer, which is applied to all input examples

    • a list of integers or a 1D tensor, with length matching the number of examples in inputs (dim 0). Each integer is applied as the target for the corresponding example.

    For outputs with > 2 dimensions, targets can be either:

    • A single tuple, which contains #output_dims - 1 elements. This target index is applied to all examples.

    • A list of tuples with length equal to the number of examples in inputs (dim 0), and each tuple containing #output_dims - 1 elements. Each tuple is applied as the target for the corresponding example.

    Default: None

  • additional_forward_args (any, optional) – If the forward function requires additional arguments other than the inputs for which attributions should not be computed, this argument can be provided. It must be either a single additional argument of a Tensor or arbitrary (non-tuple) type or a tuple containing multiple additional arguments including tensors or any arbitrary python types. These arguments are provided to forward_func in order following the arguments in inputs. For a tensor, the first dimension of the tensor must correspond to the number of examples. It will be repeated for each of n_steps along the integrated path. For all other types, the given argument is used for all forward evaluations. Note that attributions are not computed with respect to these arguments. Default: None

  • n_neighbors (int, optional) – Number of neighbors to use by default. Must be provided if it has not been set in the init. Can be an integer (same for every inputs) or a tuple. Default: None

  • n_steps (int, optional) – The number of steps used by the approximation method. Default: 5.

  • n_steiner (int, optional) – Add a certain number of steiner points into the graph. These points are added following the fixed scheme. For more information, please refer to the section 4.1.3 of https://arxiv.org/pdf/2007.10430. Default: None

  • method (string, optional) – Method for approximating the integral, one of riemann_right, riemann_left, riemann_middle, riemann_trapezoid or gausslegendre. Default: gausslegendre if no method is provided.

  • internal_batch_size (int, optional) – Divides total #steps * #examples data points into chunks of size at most internal_batch_size, which are computed (forward / backward passes) sequentially. internal_batch_size must be at least equal to #examples. For DataParallel models, each batch is split among the available devices, so evaluations on each available device contain internal_batch_size / num_devices examples. If internal_batch_size is None, then all evaluations are processed in one batch. Default: None

  • return_curvature (bool, optional) – Indicates whether to return the curvature or not. If return_curvature is set to True curvature will be returned in a tuple following attributions and optionally convergence delta. Default: False

  • return_convergence_delta (bool, optional) – Indicates whether to return convergence delta or not. If return_convergence_delta is set to True convergence delta will be returned in a tuple following attributions. Default: False

  • distance (str, optional) –

    Which distance to use with the A* algorithm:

    Default: ‘geodesic’

  • show_progress (bool, optional) – Displays the progress of computation. It will try to use tqdm if available for advanced features (e.g. time estimation). Otherwise, it will fallback to a simple output of progress. Default: False

Returns:

  • attributions (tensor or tuple of tensors):

    Integrated gradients with respect to each input feature. attributions will always be the same size as the provided inputs, with each value providing the attribution of the corresponding input index. If a single tensor is provided as inputs, a single tensor is returned. If a tuple is provided for inputs, a tuple of corresponding sized tensors is returned.

  • delta (tensor, returned if return_convergence_delta=True):

    The difference between the total approximated and true integrated gradients. This is computed using the property that the total sum of forward_func(inputs) - forward_func(baselines) must equal the total sum of the integrated gradient. Delta is calculated per example, meaning that the number of elements in returned delta tensor is equal to the number of examples in inputs.

  • curvature (tensor, returned if return_curvature=True):

    The difference between the distance along the path computed by the A* algorithm and the euclidean distance between inputs and baselines. This value, always positive, returns a measure of the curvature of the input space, with the inner product of the gradient of the model:

    \[<\nabla F(x)^T, \nabla F(x)>\]

    as a metric. A higher value indicates a higher curvature. This value however depends on the path and is as such only an indication of the true curvature of the input space.

Return type:

attributions or tuple with attributions, delta (optional) or /and curvature (optional)

static compute_curvature(inputs: TensorOrTupleOfTensorsGeneric, baselines: Tuple[Union[Tensor, int, float], ...], data: Tuple[Tensor, ...], knns: Tuple[Tensor, ...], idx: Tuple[Tensor, ...], grads_idx: Tuple[List, ...], paths_len: Tuple[List[int]]) → Tuple[Tensor, ...][source]

Compute the curvature of the input space, as the difference between the euclidean distance along the path computed by the A* algorithm and the euclidean distance between the inputs and baseline. The curvature is always positive.

has_convergence_delta() → bool[source]

This method informs the user whether the attribution algorithm provides a convergence delta (aka an approximation error) or not. Convergence delta may serve as a proxy of correctness of attribution algorithm’s approximation. If deriving attribution class provides a compute_convergence_delta method, it should override both compute_convergence_delta and has_convergence_delta methods.

Returns:

Returns whether the attribution algorithm provides a convergence delta (aka approximation error) or not.

Return type:

bool

class tint.attr.LofKernelShap(forward_func: Callable, embeddings: Tensor, n_neighbors: int = 20, **kwargs)[source]

Local Outlier Factor Kernel Shap.

This method compute a Local Outlier Factor score for every perturbed data. This score is then used to update the weight given by the similarity function:

\[new_weight(x) = similarity(x) * \frac{-1}{lof_score(x)}\]

If the perturbed data is considered more out of sample, the weight of this data will be reduced.

Parameters:

  • forward_func (Callable) – The forward function of the model or any modification of it.

  • embeddings (Tensor) – Tensor of embeddings to compute the LOF.

  • n_neighbors (int) – Number of neighbors to use by default. Default to 20

References

Time Interpret: a Unified Model Interpretability Library for Time Series

Examples

>>> import torch as th
>>> from tint.attr import LofKernelShap
>>> from tint.models import MLP

>>> inputs = th.rand(8, 7, 5)
>>> data = th.rand(32, 7, 5)
>>> mlp = MLP([5, 3, 1])

>>> explainer = LofKernelShap(mlp, data, n_neighbors=2)
>>> attr = explainer.attribute(inputs, target=0)
class tint.attr.LofLime(forward_func: Callable, embeddings: Tensor, n_neighbors: int = 20, interpretable_model: Optional[Model] = None, similarity_func: Optional[Callable] = None, perturb_func: Optional[Callable] = None, **kwargs)[source]

Local Outlier Factor Lime.

This method compute a Local Outlier Factor score for every perturbed data. This score is then used to update the weight given by the similarity function:

\[new_weight(x) = similarity(x) * \frac{-1}{lof_score(x)}\]

If the perturbed data is considered more out of sample, the weight of this data will be reduced.

Parameters:

  • forward_func (Callable) – The forward function of the model or any modification of it.

  • embeddings (Tensor) – Tensor of embeddings to compute the LOF.

  • n_neighbors (int) – Number of neighbors to use by default. Default to 20

  • interpretable_model (optional, Model) –

    Model object to train interpretable model.

    This argument is optional and defaults to SkLearnLasso(alpha=0.01), which is a wrapper around the Lasso linear model in SkLearn. This requires having sklearn version >= 0.23 available.

    Other predefined interpretable linear models are provided in captum._utils.models.linear_model.

    Alternatively, a custom model object must provide a fit method to train the model, given a dataloader, with batches containing three tensors:

    • interpretable_inputs: Tensor [2D num_samples x num_interp_features],

    • expected_outputs: Tensor [1D num_samples],

    • weights: Tensor [1D num_samples]

    The model object must also provide a representation method to access the appropriate coefficients or representation of the interpretable model after fitting.

    Note that calling fit multiple times should retrain the interpretable model, each attribution call reuses the same given interpretable model object.

  • similarity_func (optional, callable) –

    Function which takes a single sample along with its corresponding interpretable representation and returns the weight of the interpretable sample for training the interpretable model. This is often referred to as a similarity kernel.

    This argument is optional and defaults to a function which applies an exponential kernel to the consine distance between the original input and perturbed input, with a kernel width of 1.0.

    A similarity function applying an exponential kernel to cosine / euclidean distances can be constructed using the provided get_exp_kernel_similarity_function in captum.attr._core.lime.

    Alternately, a custom callable can also be provided. The expected signature of this callable is:

    >>> def similarity_func(
>>>    original_input: Tensor or tuple of Tensors,
>>>    perturbed_input: Tensor or tuple of Tensors,
>>>    perturbed_interpretable_input:
>>>        Tensor [2D 1 x num_interp_features],
>>>    **kwargs: Any
>>> ) -> float or Tensor containing float scalar

    perturbed_input and original_input will be the same type and contain tensors of the same shape, with original_input being the same as the input provided when calling attribute.

    kwargs includes baselines, feature_mask, num_interp_features (integer, determined from feature mask).

  • perturb_func (optional, callable) –

    Function which returns a single sampled input, which is a binary vector of length num_interp_features, or a generator of such tensors.

    This function is optional, the default function returns a binary vector where each element is selected independently and uniformly at random. Custom logic for selecting sampled binary vectors can be implemented by providing a function with the following expected signature:

    >>> perturb_func(
>>>    original_input: Tensor or tuple of Tensors,
>>>    **kwargs: Any
>>> ) -> Tensor [Binary 2D Tensor 1 x num_interp_features]
>>>  or generator yielding such tensors

    kwargs includes baselines, feature_mask, num_interp_features (integer, determined from feature mask).

References

Time Interpret: a Unified Model Interpretability Library for Time Series

Examples

>>> import torch as th
>>> from tint.attr import LofLime
>>> from tint.models import MLP

>>> inputs = th.rand(8, 7, 5)
>>> data = th.rand(32, 7, 5)
>>> mlp = MLP([5, 3, 1])

>>> explainer = LofLime(mlp, data, n_neighbors=2)
>>> attr = explainer.attribute(inputs, target=0)
class tint.attr.NonLinearitiesTunnel(attribution_method: Attribution)[source]

Replace non linearities (or any module) with others before running an attribution method. This tunnel is originally intended to replace ReLU activations with Softplus to smooth the explanations.

Warning

This method will break if the forward_func contains functional non linearities with additional arguments that need to be replaced. For instance, replacing F.softmax(x, dim=-1) is not possible due to the presence of the extra argument dim.

Hint

In order to replace any layer, a nn.Module must be passed as forward_func. In particular, passing model.forward will result in not replacing any layer in model.

Parameters:

attribution_method (Attribution) – An instance of any attribution algorithm of type Attribution. E.g. Integrated Gradients, Conductance or Saliency.

References

  1. Time Interpret: a Unified Model Interpretability Library for Time Series

  2. Explanations can be manipulated and geometry is to blame

Examples

>>> import torch as th
>>> from captum.attr import Saliency
>>> from tint.attr import NonLinearitiesTunnel
>>> from tint.models import MLP

>>> inputs = th.rand(8, 7, 5)
>>> mlp = MLP([5, 3, 1])

>>> explainer = NonLinearitiesTunnel(Saliency(mlp))
>>> attr = explainer.attribute(inputs, target=0)
attribute(inputs: Union[Tensor, Tuple[Tensor, ...]], to_replace: Union[Module, Tuple[Module, ...]] = ReLU(), replace_with: Union[Module, Tuple[Module, ...]] = Softplus(beta=1, threshold=20), **kwargs: Any) → Union[Tensor, Tuple[Tensor, Tensor], Tuple[Tensor, ...], Tuple[Tuple[Tensor, ...], Tensor]][source]
Parameters:

  • inputs (tensor or tuple of tensors) – Input for which integrated gradients are computed. If forward_func takes a single tensor as input, a single input tensor should be provided. If forward_func takes multiple tensors as input, a tuple of the input tensors should be provided. It is assumed that for all given input tensors, dimension 0 corresponds to the number of examples, and if multiple input tensors are provided, the examples must be aligned appropriately. It is also assumed that for all given input tensors, dimension 1 corresponds to the time dimension, and if multiple input tensors are provided, the examples must be aligned appropriately.

  • to_replace (nn.Module, tuple, optional) – Non linearities to be replaced. The linearities of type listed here will be replaced by replaced_by non linearities before running the attribution method. This can be an instance or a class. If a class is passed, default attributes are used. Default: nn.ReLU()

  • replace_with (nn.Module, tuple, optional) – Non linearities to replace the ones listed in to_replace. Default: nn.Softplus()

  • **kwargs – (Any, optional): Contains a list of arguments that are passed to attribution_method attribution algorithm. Any additional arguments that should be used for the chosen attribution method should be included here. For instance, such arguments include additional_forward_args and baselines.

Returns:

  • attributions (tensor or tuple of tensors):

    Attribution with respect to each input feature. attributions will always be the same size as the provided inputs, with each value providing the attribution of the corresponding input index. If a single tensor is provided as inputs, a single tensor is returned. If a tuple is provided for inputs, a tuple of corresponding sized tensors is returned.

  • delta (float, returned if return_convergence_delta=True):

    Approximation error computed by the attribution algorithm. Not all attribution algorithms return delta value. It is computed only for some algorithms, e.g. integrated gradients.

Return type:

attributions or 2-element tuple of attributions, delta

has_convergence_delta() → bool[source]

This method informs the user whether the attribution algorithm provides a convergence delta (aka an approximation error) or not. Convergence delta may serve as a proxy of correctness of attribution algorithm’s approximation. If deriving attribution class provides a compute_convergence_delta method, it should override both compute_convergence_delta and has_convergence_delta methods.

Returns:

Returns whether the attribution algorithm provides a convergence delta (aka approximation error) or not.

Return type:

bool

class tint.attr.Occlusion(forward_func: Callable)[source]

A perturbation based approach to compute attribution, involving replacing each contiguous rectangular region with a given baseline / reference, and computing the difference in output. For features located in multiple regions (hyperrectangles), the corresponding output differences are averaged to compute the attribution for that feature.

The first patch is applied with the corner aligned with all indices 0, and strides are applied until the entire dimension range is covered. Note that this may cause the final patch applied in a direction to be cut-off and thus smaller than the target occlusion shape.

More details regarding the occlusion (or grey-box / sliding window) method can be found in the original paper and in the DeepExplain implementation. https://arxiv.org/abs/1311.2901 https://github.com/marcoancona/DeepExplain/blob/master/deepexplain/tensorflow/methods.py#L401

Parameters:

forward_func (Callable) – The forward function of the model or any modification of it.

Examples

>>> import torch as th
>>> from tint.attr import Occlusion
>>> from tint.models import MLP

>>> inputs = th.rand(8, 7, 5)
>>> mlp = MLP([5, 3, 1])

>>> explainer = Occlusion(mlp)
>>> attr = explainer.attribute(inputs, (1, 1))
attribute(inputs: TensorOrTupleOfTensorsGeneric, sliding_window_shapes: Union[Tuple[int, ...], Tuple[Tuple[int, ...], ...]], strides: Union[None, int, Tuple[int, ...], Tuple[Union[int, Tuple[int, ...]], ...]] = None, baselines: Union[None, Tensor, int, float, Tuple[Union[Tensor, int, float], ...]] = None, target: Union[None, int, Tuple[int, ...], Tensor, List[Tuple[int, ...]], List[int]] = None, additional_forward_args: Any = None, perturbations_per_eval: int = 1, attributions_fn: Callable = None, show_progress: bool = False) → TensorOrTupleOfTensorsGeneric[source]

Attribute method.

Parameters:

  • inputs (tensor or tuple of tensors) – Input for which occlusion attributions are computed. If forward_func takes a single tensor as input, a single input tensor should be provided. If forward_func takes multiple tensors as input, a tuple of the input tensors should be provided. It is assumed that for all given input tensors, dimension 0 corresponds to the number of examples (aka batch size), and if multiple input tensors are provided, the examples must be aligned appropriately.

  • sliding_window_shapes (tuple or tuple of tuples) – Shape of patch (hyperrectangle) to occlude each input. For a single input tensor, this must be a tuple of length equal to the number of dimensions of the input tensor - 1, defining the dimensions of the patch. If the input tensor is 1-d, this should be an empty tuple. For multiple input tensors, this must be a tuple containing one tuple for each input tensor defining the dimensions of the patch for that input tensor, as described for the single tensor case.

  • strides (int or tuple or tuple of ints or tuple of tuples, optional) – This defines the step by which the occlusion hyperrectangle should be shifted by in each direction for each iteration. For a single tensor input, this can be either a single integer, which is used as the step size in each direction, or a tuple of integers matching the number of dimensions in the occlusion shape, defining the step size in the corresponding dimension. For multiple tensor inputs, this can be either a tuple of integers, one for each input tensor (used for all dimensions of the corresponding tensor), or a tuple of tuples, providing the stride per dimension for each tensor. To ensure that all inputs are covered by at least one sliding window, the stride for any dimension must be <= the corresponding sliding window dimension if the sliding window dimension is less than the input dimension. If None is provided, a stride of 1 is used for each dimension of each input tensor. Default: None

  • baselines (scalar, tensor, tuple of scalars or tensors, optional) –

    Baselines define reference value which replaces each feature when occluded. Baselines can be provided as:

    • a single tensor, if inputs is a single tensor, with exactly the same dimensions as inputs or broadcastable to match the dimensions of inputs

    • a single scalar, if inputs is a single tensor, which will be broadcasted for each input value in input tensor.

    • a tuple of tensors or scalars, the baseline corresponding to each tensor in the inputs’ tuple can be:

      • either a tensor with matching dimensions to corresponding tensor in the inputs’ tuple or the first dimension is one and the remaining dimensions match with the corresponding input tensor.

      • or a scalar, corresponding to a tensor in the inputs’ tuple. This scalar value is broadcasted for corresponding input tensor.

    In the cases when baselines is not provided, we internally use zero scalar corresponding to each input tensor. Default: None

  • target (int, tuple, tensor or list, optional) –

    Output indices for which difference is computed (for classification cases, this is usually the target class). If the network returns a scalar value per example, no target index is necessary. For general 2D outputs, targets can be either:

    • a single integer or a tensor containing a single integer, which is applied to all input examples

    • a list of integers or a 1D tensor, with length matching the number of examples in inputs (dim 0). Each integer is applied as the target for the corresponding example.

    For outputs with > 2 dimensions, targets can be either:

    • A single tuple, which contains #output_dims - 1 elements. This target index is applied to all examples.

    • A list of tuples with length equal to the number of examples in inputs (dim 0), and each tuple containing #output_dims - 1 elements. Each tuple is applied as the target for the corresponding example.

    Default: None

  • additional_forward_args (any, optional) – If the forward function requires additional arguments other than the inputs for which attributions should not be computed, this argument can be provided. It must be either a single additional argument of a Tensor or arbitrary (non-tuple) type or a tuple containing multiple additional arguments including tensors or any arbitrary python types. These arguments are provided to forward_func in order following the arguments in inputs. For a tensor, the first dimension of the tensor must correspond to the number of examples. For all other types, the given argument is used for all forward evaluations. Note that attributions are not computed with respect to these arguments. Default: None

  • perturbations_per_eval (int, optional) – Allows multiple occlusions to be included in one batch (one call to forward_fn). By default, perturbations_per_eval is 1, so each occlusion is processed individually. Each forward pass will contain a maximum of perturbations_per_eval * #examples samples. For DataParallel models, each batch is split among the available devices, so evaluations on each available device contain at most (perturbations_per_eval * #examples) / num_devices samples. Default: 1

  • attributions_fn (Callable, optional) – Applies a function to the attributions before performing the weighted sum. Default: None

  • show_progress (bool, optional) – Displays the progress of computation. It will try to use tqdm if available for advanced features (e.g. time estimation). Otherwise, it will fallback to a simple output of progress. Default: False

Returns:

  • attributions (tensor or tuple of tensors):

    The attributions with respect to each input feature. Attributions will always be the same size as the provided inputs, with each value providing the attribution of the corresponding input index. If a single tensor is provided as inputs, a single tensor is returned. If a tuple is provided for inputs, a tuple of corresponding sized tensors is returned.

Return type:

tensor or tuple of tensors of attributions

Examples

>>> # SimpleClassifier takes a single input tensor of size Nx4x4,
>>> # and returns an Nx3 tensor of class probabilities.
>>> net = SimpleClassifier()
>>> # Generating random input with size 2 x 4 x 4
>>> input = torch.randn(2, 4, 4)
>>> # Defining Occlusion interpreter
>>> ablator = Occlusion(net)
>>> # Computes occlusion attribution, ablating each 3x3 patch,
>>> # shifting in each direction by the default of 1.
>>> attr = ablator.attribute(input, target=1, sliding_window_shapes=(3,3))
class tint.attr.Retain(forward_func: Optional[Callable] = None, retain: Optional[RetainNet] = None, datamodule: Optional[LightningDataModule] = None, features: Optional[Tensor] = None, labels: Optional[Tensor] = None, trainer: Optional[Trainer] = None, batch_size: int = 32)[source]

Retain explainer method.

Parameters:

  • forward_func (Callable) – The forward function of the model or any modification of it.

  • retain (RetainNet) – A Retain network as a Pytorch Lightning module. If None, a default Retain Net will be created. Default to None

  • datamodule (LightningDataModule) – A Pytorch Lightning data module which will be used to train the RetainNet. Either a datamodule or features must be provided, they cannot be None together. Default to None

  • features (Tensor) – A tensor of features which will be used to train the RetainNet. Either a datamodule or features must be provided, they cannot be None together. If both are provided, features is ignored. Default to None

References

RETAIN: An Interpretable Predictive Model for Healthcare using Reverse Time Attention Mechanism

Examples

>>> import torch as th
>>> from tint.attr import Retain

>>> inputs = th.rand(8, 7, 5)
>>> data = th.rand(32, 7, 5)
>>> labels = th.randint(2, (32, 7))

>>> explainer = Retain(features=data, labels=labels)
>>> attr = explainer.attribute(inputs, target=th.randint(2, (8, 7))))
attribute(inputs: TensorOrTupleOfTensorsGeneric, target: Union[None, int, Tuple[int, ...], Tensor, List[Tuple[int, ...]], List[int]], return_temporal_attributions: bool = False) → TensorOrTupleOfTensorsGeneric[source]

attribute method.

Parameters:

  • inputs (tensor or tuple of tensors) – Input for which integrated gradients are computed. If forward_func takes a single tensor as input, a single input tensor should be provided. If forward_func takes multiple tensors as input, a tuple of the input tensors should be provided. It is assumed that for all given input tensors, dimension 0 corresponds to the number of examples, and if multiple input tensors are provided, the examples must be aligned appropriately.

  • target (int, tuple, tensor or list, optional) –

    Output indices for which gradients are computed (for classification cases, this is usually the target class). If the network returns a scalar value per example, no target index is necessary. For general 2D outputs, targets can be either:

    • a single integer or a tensor containing a single integer, which is applied to all input examples

    • a list of integers or a 1D tensor, with length matching the number of examples in inputs (dim 0). Each integer is applied as the target for the corresponding example.

    For outputs with > 2 dimensions, targets can be either:

    • A single tuple, which contains #output_dims - 1 elements. This target index is applied to all examples.

    • A list of tuples with length equal to the number of examples in inputs (dim 0), and each tuple containing #output_dims - 1 elements. Each tuple is applied as the target for the corresponding example.

    Default: None

  • return_temporal_attributions (bool) – Whether to return attributions for all times or not. Default: False

Returns:

The attributions with respect to each input feature. Attributions will always be the same size as the provided inputs, with each value providing the attribution of the corresponding input index. If a single tensor is provided as inputs, a single tensor is returned. If a tuple is provided for inputs, a tuple of corresponding sized tensors is returned.

Return type:

  • attributions (tensor or tuple of tensors)

representation(inputs: Tensor, target: Optional[Tensor] = None)[source]

Get representations based on a model, inputs and potentially targets.

Parameters:

  • inputs (th.Tensor) – Input data.

  • target (th.Tensor) – Targets. Default to None

Returns:

attributions.

Return type:

th.Tensor

class tint.attr.SequentialIntegratedGradients(forward_func: Callable, multiply_by_inputs: bool = True)[source]

Sequential Integrated Gradients.

This method is the regular Integrated Gradients (IG) applied on each component of a sequence. However, the baseline is specific to each component: it keeps fixed the rest of the sequence while only setting the component of interest to a reference baseline.

For instance, on a setence of m words, the attribution of each word is computed by running IG with a specific baseline: fixing every other word to their current value, and replacing the word of interest with “<pad>”, an uninformative baseline.

This method can be computationally expensive on long sequences, as it needs to compute IG on each component individually. It is therefore suggested to reduce n_steps when using this method on long sequences.

Parameters:

  • forward_func (callable) – The forward function of the model or any modification of it

  • multiply_by_inputs (bool, optional) –

    Indicates whether to factor model inputs’ multiplier in the final attribution scores. In the literature this is also known as local vs global attribution. If inputs’ multiplier isn’t factored in, then that type of attribution method is also called local attribution. If it is, then that type of attribution method is called global. More detailed can be found here: https://arxiv.org/abs/1711.06104

    In case of integrated gradients, if multiply_by_inputs is set to True, final sensitivity scores are being multiplied by (inputs - baselines).

References

Sequential Integrated Gradients: a simple but effective method for explaining language models

Examples

>>> import torch as th
>>> from tint.attr import SequentialIntegratedGradients
>>> from tint.models import MLP

>>> inputs = th.rand(8, 7, 5)
>>> mlp = MLP([5, 3, 1])

>>> explainer = SequentialIntegratedGradients(mlp)
>>> attr = explainer.attribute(inputs, target=0)
attribute(inputs: TensorOrTupleOfTensorsGeneric, baselines: BaselineType = None, target: TargetType = None, additional_forward_args: Any = None, n_steps: int = 50, method: str = 'gausslegendre', internal_batch_size: Union[None, int] = None, return_convergence_delta: Literal[False] = False, show_progress: bool = False) → TensorOrTupleOfTensorsGeneric[source]
attribute(inputs: TensorOrTupleOfTensorsGeneric, baselines: BaselineType = None, target: TargetType = None, additional_forward_args: Any = None, n_steps: int = 50, method: str = 'gausslegendre', internal_batch_size: Union[None, int] = None, *, return_convergence_delta: Literal[True], show_progress: bool = False) → Tuple[TensorOrTupleOfTensorsGeneric, Tensor]

This method attributes the output of the model with given target index (in case it is provided, otherwise it assumes that output is a scalar) to the inputs of the model using the approach described above.

In addition to that it also returns, if return_convergence_delta is set to True, integral approximation delta based on the completeness property of integrated gradients.

Parameters:

  • inputs (tensor or tuple of tensors) – Input for which integrated gradients are computed. If forward_func takes a single tensor as input, a single input tensor should be provided. If forward_func takes multiple tensors as input, a tuple of the input tensors should be provided. It is assumed that for all given input tensors, dimension 0 corresponds to the number of examples, and if multiple input tensors are provided, the examples must be aligned appropriately.

  • baselines (scalar, tensor, tuple of scalars or tensors, optional) –

    Baselines define the starting point from which integral is computed and can be provided as:

    • a single tensor, if inputs is a single tensor, with exactly the same dimensions as inputs or the first dimension is one and the remaining dimensions match with inputs.

    • a single scalar, if inputs is a single tensor, which will be broadcasted for each input value in input tensor.

    • a tuple of tensors or scalars, the baseline corresponding to each tensor in the inputs’ tuple can be:

      • either a tensor with matching dimensions to corresponding tensor in the inputs’ tuple or the first dimension is one and the remaining dimensions match with the corresponding input tensor.

      • or a scalar, corresponding to a tensor in the inputs’ tuple. This scalar value is broadcasted for corresponding input tensor.

    In the cases when baselines is not provided, we internally use zero scalar corresponding to each input tensor.

    Default: None

  • target (int, tuple, tensor or list, optional) –

    Output indices for which gradients are computed (for classification cases, this is usually the target class). If the network returns a scalar value per example, no target index is necessary. For general 2D outputs, targets can be either:

    • a single integer or a tensor containing a single integer, which is applied to all input examples

    • a list of integers or a 1D tensor, with length matching the number of examples in inputs (dim 0). Each integer is applied as the target for the corresponding example.

    For outputs with > 2 dimensions, targets can be either:

    • A single tuple, which contains #output_dims - 1 elements. This target index is applied to all examples.

    • A list of tuples with length equal to the number of examples in inputs (dim 0), and each tuple containing #output_dims - 1 elements. Each tuple is applied as the target for the corresponding example.

    Default: None

  • additional_forward_args (any, optional) – If the forward function requires additional arguments other than the inputs for which attributions should not be computed, this argument can be provided. It must be either a single additional argument of a Tensor or arbitrary (non-tuple) type or a tuple containing multiple additional arguments including tensors or any arbitrary python types. These arguments are provided to forward_func in order following the arguments in inputs. For a tensor, the first dimension of the tensor must correspond to the number of examples. It will be repeated for each of n_steps along the integrated path. For all other types, the given argument is used for all forward evaluations. Note that attributions are not computed with respect to these arguments. Default: None

  • n_steps (int, optional) – The number of steps used by the approximation method. Default: 50.

  • method (string, optional) – Method for approximating the integral, one of riemann_right, riemann_left, riemann_middle, riemann_trapezoid or gausslegendre. Default: gausslegendre if no method is provided.

  • internal_batch_size (int, optional) – Divides total #steps * #examples data points into chunks of size at most internal_batch_size, which are computed (forward / backward passes) sequentially. internal_batch_size must be at least equal to #examples. For DataParallel models, each batch is split among the available devices, so evaluations on each available device contain internal_batch_size / num_devices examples. If internal_batch_size is None, then all evaluations are processed in one batch. Default: None

  • return_convergence_delta (bool, optional) – Indicates whether to return convergence delta or not. If return_convergence_delta is set to True convergence delta will be returned in a tuple following attributions. Default: False

  • show_progress (bool, optional) – Displays the progress of computation. It will try to use tqdm if available for advanced features (e.g. time estimation). Otherwise, it will fallback to a simple output of progress. Default: False

Returns:

  • attributions (tensor or tuple of tensors):

    Integrated gradients with respect to each input feature. attributions will always be the same size as the provided inputs, with each value providing the attribution of the corresponding input index. If a single tensor is provided as inputs, a single tensor is returned. If a tuple is provided for inputs, a tuple of corresponding sized tensors is returned.

  • delta (tensor, returned if return_convergence_delta=True):

    The difference between the total approximated and true integrated gradients. This is computed using the property that the total sum of forward_func(inputs) - forward_func(baselines) must equal the total sum of the integrated gradient. Delta is calculated per example, meaning that the number of elements in returned delta tensor is equal to the number of of examples in inputs.

Return type:

attributions or 2-element tuple of attributions, delta

Examples:

>>> # ImageClassifier takes a single input tensor of images Nx3x32x32,
>>> # and returns an Nx10 tensor of class probabilities.
>>> net = ImageClassifier()
>>> sig = SequentialIntegratedGradients(net)
>>> input = torch.randn(2, 3, 32, 32, requires_grad=True)
>>> # Computes integrated gradients for class 3.
>>> attribution = sig.attribute(input, target=3)
has_convergence_delta() → bool[source]

This method informs the user whether the attribution algorithm provides a convergence delta (aka an approximation error) or not. Convergence delta may serve as a proxy of correctness of attribution algorithm’s approximation. If deriving attribution class provides a compute_convergence_delta method, it should override both compute_convergence_delta and has_convergence_delta methods.

Returns:

Returns whether the attribution algorithm provides a convergence delta (aka approximation error) or not.

Return type:

bool

class tint.attr.TemporalAugmentedOcclusion(forward_func: Callable, data: TensorOrTupleOfTensorsGeneric, n_sampling: int = 1, is_temporal: bool = False)[source]

Temporal Augmented Occlusion.

This method modifies the original augmented occlusion by only perturbing the last time, leaving the previous times unchanged. It can be used together with time_forward_tunnel to compute attributions on time series.

Parameters:

  • forward_func (callable) – The forward function of the model or any modification of it

  • data (tuple, Tensor) – The data from which the baselines are sampled.

  • n_sampling (int) – Number of sampling to run for each occlusion. Default to 1

  • is_temporal (bool) – Whether the data is temporal or not. If True, the data will be ablated to the inputs on the temporal dimension (dimension 1). Default to False

References

What went wrong and when? Instance-wise Feature Importance for Time-series Models

Examples

>>> import torch as th
>>> from tint.attr import TemporalAugmentedOcclusion
>>> from tint.models import MLP

>>> inputs = th.rand(8, 7, 5)
>>> data = th.rand(32, 7, 5)
>>> mlp = MLP([5, 3, 1])

>>> explainer = TemporalAugmentedOcclusion(mlp, data)
>>> attr = explainer.attribute(inputs, (1,))
attribute(inputs: TensorOrTupleOfTensorsGeneric, sliding_window_shapes: Union[Tuple[int, ...], Tuple[Tuple[int, ...], ...]], strides: Union[None, int, Tuple[int, ...], Tuple[Union[int, Tuple[int, ...]], ...]] = None, target: Union[None, int, Tuple[int, ...], Tensor, List[Tuple[int, ...]], List[int]] = None, additional_forward_args: Any = None, perturbations_per_eval: int = 1, attributions_fn: Callable = None, show_progress: bool = False) → TensorOrTupleOfTensorsGeneric[source]
Parameters:

  • inputs (tensor or tuple of tensors) – Input for which occlusion attributions are computed. If forward_func takes a single tensor as input, a single input tensor should be provided. If forward_func takes multiple tensors as input, a tuple of the input tensors should be provided. It is assumed that for all given input tensors, dimension 0 corresponds to the number of examples (aka batch size), and if multiple input tensors are provided, the examples must be aligned appropriately.

  • sliding_window_shapes (tuple or tuple of tuples) – Shape of patch (hyperrectangle) to occlude each input. For a single input tensor, this must be a tuple of length equal to the number of dimensions of the input tensor - 2, defining the dimensions of the patch. If the input tensor is 2-d, this should be an empty tuple. For multiple input tensors, this must be a tuple containing one tuple for each input tensor defining the dimensions of the patch for that input tensor, as described for the single tensor case.

  • strides (int or tuple or tuple of ints or tuple of tuples, optional) – This defines the step by which the occlusion hyperrectangle should be shifted by in each direction for each iteration. For a single tensor input, this can be either a single integer, which is used as the step size in each direction, or a tuple of integers matching the number of dimensions in the occlusion shape, defining the step size in the corresponding dimension. For multiple tensor inputs, this can be either a tuple of integers, one for each input tensor (used for all dimensions of the corresponding tensor), or a tuple of tuples, providing the stride per dimension for each tensor. To ensure that all inputs are covered by at least one sliding window, the stride for any dimension must be <= the corresponding sliding window dimension if the sliding window dimension is less than the input dimension. If None is provided, a stride of 1 is used for each dimension of each input tensor. Default: None

  • target (int, tuple, tensor or list, optional) –

    Output indices for which difference is computed (for classification cases, this is usually the target class). If the network returns a scalar value per example, no target index is necessary. For general 2D outputs, targets can be either:

    • a single integer or a tensor containing a single integer, which is applied to all input examples

    • a list of integers or a 1D tensor, with length matching the number of examples in inputs (dim 0). Each integer is applied as the target for the corresponding example.

    For outputs with > 2 dimensions, targets can be either:

    • A single tuple, which contains #output_dims - 1 elements. This target index is applied to all examples.

    • A list of tuples with length equal to the number of examples in inputs (dim 0), and each tuple containing #output_dims - 1 elements. Each tuple is applied as the target for the corresponding example.

    Default: None

  • additional_forward_args (any, optional) – If the forward function requires additional arguments other than the inputs for which attributions should not be computed, this argument can be provided. It must be either a single additional argument of a Tensor or arbitrary (non-tuple) type or a tuple containing multiple additional arguments including tensors or any arbitrary python types. These arguments are provided to forward_func in order following the arguments in inputs. For a tensor, the first dimension of the tensor must correspond to the number of examples. For all other types, the given argument is used for all forward evaluations. Note that attributions are not computed with respect to these arguments. Default: None

  • perturbations_per_eval (int, optional) – Allows multiple occlusions to be included in one batch (one call to forward_fn). By default, perturbations_per_eval is 1, so each occlusion is processed individually. Each forward pass will contain a maximum of perturbations_per_eval * #examples samples. For DataParallel models, each batch is split among the available devices, so evaluations on each available device contain at most (perturbations_per_eval * #examples) / num_devices samples. Default: 1

  • attributions_fn (Callable, optional) – Applies a function to the attributions before performing the weighted sum. Default: None

  • show_progress (bool, optional) – Displays the progress of computation. It will try to use tqdm if available for advanced features (e.g. time estimation). Otherwise, it will fallback to a simple output of progress. Default: False

Returns:

  • attributions (tensor or tuple of tensors):

    The attributions with respect to each input feature. Attributions will always be the same size as the provided inputs, with each value providing the attribution of the corresponding input index. If a single tensor is provided as inputs, a single tensor is returned. If a tuple is provided for inputs, a tuple of corresponding sized tensors is returned.

Return type:

tensor or tuple of tensors of attributions

class tint.attr.TemporalIntegratedGradients(forward_func: Callable, multiply_by_inputs: bool = True)[source]

Temporal Integrated Gradients.

This method computes gradients iteratively on a time series as such: it crops the sequence up to a time, and only moves this last time from a baseline to its original value.

The number of steps per time depends on the strategy. If it is 'fixed', then n_steps gradients are computed for each time. If it is 'interval', the number of steps depends on the interval between two times: the larger, the greater number of points.

Parameters:

  • forward_func (callable) – The forward function of the model or any modification of it.

  • multiply_by_inputs (bool, optional) –

    Indicates whether to factor model inputs’ multiplier in the final attribution scores. In the literature this is also known as local vs global attribution. If inputs’ multiplier isn’t factored in, then that type of attribution method is also called local attribution. If it is, then that type of attribution method is called global. More detailed can be found here: https://arxiv.org/abs/1711.06104

    In case of integrated gradients, if multiply_by_inputs is set to True, final sensitivity scores are being multiplied by (inputs - baselines).

References

Time Interpret: a Unified Model Interpretability Library for Time Series

Examples

>>> import torch as th
>>> from tint.attr import TemporalIntegratedGradients
>>> from tint.models import MLP

>>> inputs = th.rand(8, 7, 5)
>>> mlp = MLP([5, 3, 1])

>>> explainer = TemporalIntegratedGradients(mlp)
>>> attr = explainer.attribute(inputs, target=0)
attribute(inputs: TensorOrTupleOfTensorsGeneric, baselines: BaselineType = None, target: TargetType = None, additional_forward_args: Any = None, times: Tensor = None, n_steps: int = 50, method: str = 'gausslegendre', strategy: str = 'fixed', internal_batch_size: Union[None, int] = None, return_convergence_delta: Literal[False] = False, temporal_target: bool = False, temporal_additional_forward_args: Tuple[bool] = None, return_temporal_attributions: bool = False, show_progress: bool = False) → TensorOrTupleOfTensorsGeneric[source]
attribute(inputs: TensorOrTupleOfTensorsGeneric, baselines: BaselineType = None, target: TargetType = None, additional_forward_args: Any = None, times: Tensor = None, n_steps: int = 50, method: str = 'gausslegendre', strategy: str = 'fixed', internal_batch_size: Union[None, int] = None, *, return_convergence_delta: Literal[True] = True, temporal_target: bool = False, temporal_additional_forward_args: Tuple[bool] = None, return_temporal_attributions: bool = False, show_progress: bool = False) → Tuple[TensorOrTupleOfTensorsGeneric, Tensor]

Attribute method.

Parameters:

  • inputs (tensor or tuple of tensors) – Input for which integrated gradients are computed. If forward_func takes a single tensor as input, a single input tensor should be provided. If forward_func takes multiple tensors as input, a tuple of the input tensors should be provided. It is assumed that for all given input tensors, dimension 0 corresponds to the number of examples, and if multiple input tensors are provided, the examples must be aligned appropriately.

  • baselines (scalar, tensor, tuple of scalars or tensors, optional) –

    Baselines define the starting point from which integral is computed and can be provided as:

    • a single tensor, if inputs is a single tensor, with exactly the same dimensions as inputs or the first dimension is one and the remaining dimensions match with inputs.

    • a single scalar, if inputs is a single tensor, which will be broadcasted for each input value in input tensor.

    • a tuple of tensors or scalars, the baseline corresponding to each tensor in the inputs’ tuple can be:

      • either a tensor with matching dimensions to corresponding tensor in the inputs’ tuple or the first dimension is one and the remaining dimensions match with the corresponding input tensor.

      • or a scalar, corresponding to a tensor in the inputs’ tuple. This scalar value is broadcasted for corresponding input tensor.

    In the cases when baselines is not provided, we internally use zero scalar corresponding to each input tensor.

    Default: None

  • target (int, tuple, tensor or list, optional) –

    Output indices for which gradients are computed (for classification cases, this is usually the target class). If the network returns a scalar value per example, no target index is necessary. For general 2D outputs, targets can be either:

    • a single integer or a tensor containing a single integer, which is applied to all input examples

    • a list of integers or a 1D tensor, with length matching the number of examples in inputs (dim 0). Each integer is applied as the target for the corresponding example.

    For outputs with > 2 dimensions, targets can be either:

    • A single tuple, which contains #output_dims - 1 elements. This target index is applied to all examples.

    • A list of tuples with length equal to the number of examples in inputs (dim 0), and each tuple containing #output_dims - 1 elements. Each tuple is applied as the target for the corresponding example.

    Default: None

  • additional_forward_args (any, optional) – If the forward function requires additional arguments other than the inputs for which attributions should not be computed, this argument can be provided. It must be either a single additional argument of a Tensor or arbitrary (non-tuple) type or a tuple containing multiple additional arguments including tensors or any arbitrary python types. These arguments are provided to forward_func in order following the arguments in inputs. For a tensor, the first dimension of the tensor must correspond to the number of examples. It will be repeated for each of n_steps along the integrated path. For all other types, the given argument is used for all forward evaluations. Note that attributions are not computed with respect to these arguments. Default: None

  • times (Tensor, optional) – Tensor of times. If not provided, it is assumed that the points are temporally equally spaced. Default: None

  • n_steps (int, optional) – The number of steps used by the approximation method. Default: 50.

  • method (string, optional) – Method for approximating the integral, one of riemann_right, riemann_left, riemann_middle, riemann_trapezoid or gausslegendre. Default: gausslegendre if no method is provided.

  • strategy (str, optinal) – Strategy to distribute gradients evaluations over time. Either 'fixed' or 'interval' Default: ‘fixed’

  • internal_batch_size (int, optional) – Divides total #steps * #examples data points into chunks of size at most internal_batch_size, which are computed (forward / backward passes) sequentially. internal_batch_size must be at least equal to #examples. For DataParallel models, each batch is split among the available devices, so evaluations on each available device contain internal_batch_size / num_devices examples. If internal_batch_size is None, then all evaluations are processed in one batch. Default: None

  • return_convergence_delta (bool, optional) – Indicates whether to return convergence delta or not. If return_convergence_delta is set to True convergence delta will be returned in a tuple following attributions. Default: False

  • temporal_target – Temporal target. Default: False

  • temporal_additional_forward_args (tuple) – Set each additional forward arg which is temporal. Only used with return_temporal_attributions. Default: None

  • return_temporal_attributions (bool) – Whether to return all saliencies for all time points or only the last one per time point. Default: False

  • show_progress (bool, optional) – Displays the progress of computation. It will try to use tqdm if available for advanced features (e.g. time estimation). Otherwise, it will fallback to a simple output of progress. Default: False

Returns:

  • attributions (tensor or tuple of tensors):

    Integrated gradients with respect to each input feature. attributions will always be the same size as the provided inputs, with each value providing the attribution of the corresponding input index. If a single tensor is provided as inputs, a single tensor is returned. If a tuple is provided for inputs, a tuple of corresponding sized tensors is returned.

  • delta (tensor, returned if return_convergence_delta=True):

    The difference between the total approximated and true integrated gradients. This is computed using the property that the total sum of forward_func(inputs) - forward_func(baselines) must equal the total sum of the integrated gradient. Delta is calculated per example, meaning that the number of elements in returned delta tensor is equal to the number of of examples in inputs.

Return type:

attributions or 2-element tuple of attributions, delta

class tint.attr.TemporalOcclusion(forward_func: Callable)[source]

Temporal Occlusion.

This method modifies the original occlusion by only perturbing the last time, leaving the previous times unchanged. It can be used together with time_forward_tunnel to compute attributions on time series.

Parameters:

forward_func (callable) – The forward function of the model or any modification of it.

References

What went wrong and when? Instance-wise Feature Importance for Time-series Models

Examples

>>> import torch as th
>>> from tint.attr import TemporalOcclusion
>>> from tint.models import MLP

>>> inputs = th.rand(8, 7, 5)
>>> mlp = MLP([5, 3, 1])

>>> explainer = TemporalOcclusion(mlp)
>>> attr = explainer.attribute(inputs, (1,))
attribute(inputs: TensorOrTupleOfTensorsGeneric, sliding_window_shapes: Union[Tuple[int, ...], Tuple[Tuple[int, ...], ...]], strides: Union[None, int, Tuple[int, ...], Tuple[Union[int, Tuple[int, ...]], ...]] = None, baselines: Union[None, Tensor, int, float, Tuple[Union[Tensor, int, float], ...]] = None, target: Union[None, int, Tuple[int, ...], Tensor, List[Tuple[int, ...]], List[int]] = None, additional_forward_args: Any = None, perturbations_per_eval: int = 1, attributions_fn: Callable = None, show_progress: bool = False) → TensorOrTupleOfTensorsGeneric[source]
Parameters:

  • inputs (tensor or tuple of tensors) – Input for which occlusion attributions are computed. If forward_func takes a single tensor as input, a single input tensor should be provided. If forward_func takes multiple tensors as input, a tuple of the input tensors should be provided. It is assumed that for all given input tensors, dimension 0 corresponds to the number of examples (aka batch size), and if multiple input tensors are provided, the examples must be aligned appropriately.

  • sliding_window_shapes (tuple or tuple of tuples) – Shape of patch (hyperrectangle) to occlude each input. For a single input tensor, this must be a tuple of length equal to the number of dimensions of the input tensor - 2, defining the dimensions of the patch. If the input tensor is 2-d, this should be an empty tuple. For multiple input tensors, this must be a tuple containing one tuple for each input tensor defining the dimensions of the patch for that input tensor, as described for the single tensor case.

  • strides (int or tuple or tuple of ints or tuple of tuples, optional) – This defines the step by which the occlusion hyperrectangle should be shifted by in each direction for each iteration. For a single tensor input, this can be either a single integer, which is used as the step size in each direction, or a tuple of integers matching the number of dimensions in the occlusion shape, defining the step size in the corresponding dimension. For multiple tensor inputs, this can be either a tuple of integers, one for each input tensor (used for all dimensions of the corresponding tensor), or a tuple of tuples, providing the stride per dimension for each tensor. To ensure that all inputs are covered by at least one sliding window, the stride for any dimension must be <= the corresponding sliding window dimension if the sliding window dimension is less than the input dimension. If None is provided, a stride of 1 is used for each dimension of each input tensor. Default: None

  • baselines (scalar, tensor, tuple of scalars or tensors, optional) –

    Baselines define reference value which replaces each feature when occluded. Baselines can be provided as:

    • a single tensor, if inputs is a single tensor, with exactly the same dimensions as inputs or broadcastable to match the dimensions of inputs

    • a single scalar, if inputs is a single tensor, which will be broadcasted for each input value in input tensor.

    • a tuple of tensors or scalars, the baseline corresponding to each tensor in the inputs’ tuple can be:

      • either a tensor with matching dimensions to corresponding tensor in the inputs’ tuple or the first dimension is one and the remaining dimensions match with the corresponding input tensor.

      • or a scalar, corresponding to a tensor in the inputs’ tuple. This scalar value is broadcasted for corresponding input tensor.

    In the cases when baselines is not provided, we internally use zero scalar corresponding to each input tensor. Default: None

  • target (int, tuple, tensor or list, optional) –

    Output indices for which difference is computed (for classification cases, this is usually the target class). If the network returns a scalar value per example, no target index is necessary. For general 2D outputs, targets can be either:

    • a single integer or a tensor containing a single integer, which is applied to all input examples

    • a list of integers or a 1D tensor, with length matching the number of examples in inputs (dim 0). Each integer is applied as the target for the corresponding example.

    For outputs with > 2 dimensions, targets can be either:

    • A single tuple, which contains #output_dims - 1 elements. This target index is applied to all examples.

    • A list of tuples with length equal to the number of examples in inputs (dim 0), and each tuple containing #output_dims - 1 elements. Each tuple is applied as the target for the corresponding example.

    Default: None

  • additional_forward_args (any, optional) – If the forward function requires additional arguments other than the inputs for which attributions should not be computed, this argument can be provided. It must be either a single additional argument of a Tensor or arbitrary (non-tuple) type or a tuple containing multiple additional arguments including tensors or any arbitrary python types. These arguments are provided to forward_func in order following the arguments in inputs. For a tensor, the first dimension of the tensor must correspond to the number of examples. For all other types, the given argument is used for all forward evaluations. Note that attributions are not computed with respect to these arguments. Default: None

  • perturbations_per_eval (int, optional) – Allows multiple occlusions to be included in one batch (one call to forward_fn). By default, perturbations_per_eval is 1, so each occlusion is processed individually. Each forward pass will contain a maximum of perturbations_per_eval * #examples samples. For DataParallel models, each batch is split among the available devices, so evaluations on each available device contain at most (perturbations_per_eval * #examples) / num_devices samples. Default: 1

  • attributions_fn (Callable, optional) – Applies a function to the attributions before performing the weighted sum. Default: None

  • show_progress (bool, optional) – Displays the progress of computation. It will try to use tqdm if available for advanced features (e.g. time estimation). Otherwise, it will fallback to a simple output of progress. Default: False

Returns:

  • attributions (tensor or tuple of tensors):

    The attributions with respect to each input feature. Attributions will always be the same size as the provided inputs, with each value providing the attribution of the corresponding input index. If a single tensor is provided as inputs, a single tensor is returned. If a tuple is provided for inputs, a tuple of corresponding sized tensors is returned.

Return type:

tensor or tuple of tensors of attributions

class tint.attr.TimeForwardTunnel(attribution_method: Attribution)[source]

Time Forward Tunnel.

Performs interpretation method by iteratively retrieving the input data up to a time, and computing the predictions using this data and the forward_func.

The true target can be passed, otherwise it will be inferred depending on the task.

Parameters:

attribution_method (Attribution) – An instance of any attribution algorithm of type Attribution. E.g. Integrated Gradients, Conductance or Saliency.

References

  1. What went wrong and when? Instance-wise Feature Importance for Time-series Models

  2. Time Interpret: a Unified Model Interpretability Library for Time Series

Examples

>>> import torch as th
>>> from captum.attr import Saliency
>>> from tint.attr import TimeForwardTunnel
>>> from tint.models import MLP

>>> inputs = th.rand(8, 7, 5)
>>> mlp = MLP([5, 3, 1])

>>> explainer = TimeForwardTunnel(Saliency(mlp))
>>> attr = explainer.attribute(inputs, target=0)
attribute(inputs: Union[Tensor, Tuple[Tensor, ...]], task: str = 'none', threshold: float = 0.5, temporal_target: bool = False, temporal_additional_forward_args: Tuple[bool] = None, return_temporal_attributions: bool = False, show_progress: bool = False, **kwargs: Any) → Union[Tensor, Tuple[Tensor, Tensor], Tuple[Tensor, ...], Tuple[Tuple[Tensor, ...], Tensor]][source]
Parameters:

  • inputs (tensor or tuple of tensors) – Input for which integrated gradients are computed. If forward_func takes a single tensor as input, a single input tensor should be provided. If forward_func takes multiple tensors as input, a tuple of the input tensors should be provided. It is assumed that for all given input tensors, dimension 0 corresponds to the number of examples, and if multiple input tensors are provided, the examples must be aligned appropriately. It is also assumed that for all given input tensors, dimension 1 corresponds to the time dimension, and if multiple input tensors are provided, the examples must be aligned appropriately.

  • task (str) – Type of task done by the model. Either 'binary', 'multilabel', 'multiclass' or 'regression'. Default: ‘binary’

  • threshold (float) – Threshold for the multilabel task. Default: 0.5

  • temporal_target (bool, optional) – Determine if the targe is temporal and needs to be cut. Default: False

  • temporal_additional_forward_args (tuple, optional) – For each additional forward arg, determine if it is temporal or not. Default: None

  • return_temporal_attributions (bool) – Whether to return all saliencies for all time points or only the last one per time point. Default: False

  • show_progress (bool, optional) – Displays the progress of computation. It will try to use tqdm if available for advanced features (e.g. time estimation). Otherwise, it will fallback to a simple output of progress. Default: False

  • **kwargs – (Any, optional): Contains a list of arguments that are passed to attribution_method attribution algorithm. Any additional arguments that should be used for the chosen attribution method should be included here. For instance, such arguments include additional_forward_args and baselines.

Returns:

  • attributions (tensor or tuple of tensors):

    Attribution with respect to each input feature. attributions will always be the same size as the provided inputs, with each value providing the attribution of the corresponding input index. If a single tensor is provided as inputs, a single tensor is returned. If a tuple is provided for inputs, a tuple of corresponding sized tensors is returned.

  • delta (float, returned if return_convergence_delta=True):

    Approximation error computed by the attribution algorithm. Not all attribution algorithms return delta value. It is computed only for some algorithms, e.g. integrated gradients.

Return type:

attributions or 2-element tuple of attributions, delta

has_convergence_delta() → bool[source]

This method informs the user whether the attribution algorithm provides a convergence delta (aka an approximation error) or not. Convergence delta may serve as a proxy of correctness of attribution algorithm’s approximation. If deriving attribution class provides a compute_convergence_delta method, it should override both compute_convergence_delta and has_convergence_delta methods.

Returns:

Returns whether the attribution algorithm provides a convergence delta (aka approximation error) or not.

Return type:

bool