Captum.ai

CaptumModel Interpretability for PyTorchIntroductionGet StartedTutorialsCheck it out in the intro videoKey FeaturesMulti-Modal Supports interpretability of models across modalities including vision, text, and more. Built on PyTorch Supports most types of PyTorch models and can be used with minimal modification to the original neural network. Extensible Open source, generic library for interpretability research. Easily implement and benchmark new algorithms. Get StartedInstall Captum:via conda (recommended):conda install captum -c pytorch Copy via pip:pip install captum Copy Create and prepare model:import numpy as np import torch import torch.nn as nn from captum.attr import IntegratedGradients class ToyModel(nn.Module): def __init__(self): super().__init__() self.lin1 = nn.Linear(3, 3) self.relu = nn.ReLU() self.lin2 = nn.Linear(3, 2) # initialize weights and biases self.lin1.weight = nn.Parameter(torch.arange(-4.0, 5.0).view(3, 3)) self.lin1.bias = nn.Parameter(torch.zeros(1,3)) self.lin2.weight = nn.Parameter(torch.arange(-3.0, 3.0).view(2, 3)) self.lin2.bias = nn.Parameter(torch.ones(1,2)) def forward(self, input): return self.lin2(self.relu(self.lin1(input))) model = ToyModel() model.eval() Copy To make computations deterministic, let's fix random seeds:torch.manual_seed(123) np.random.seed(123) Copy Define input and baseline tensors:input = torch.rand(2, 3) baseline = torch.zeros(2, 3) Copy Select algorithm to instantiate and apply (Integrated Gradients in this example):ig = IntegratedGradients(model) attributions, delta = ig.attribute(input, baseline, target=0, return_convergence_delta=True) print('IG Attributions:', attributions) print('Convergence Delta:', delta) Copy View Output:IG Attributions: tensor([[-0.5922, -1.5497, -1.0067], [ 0.0000, -0.2219, -5.1991]]) Convergence Delta: tensor([2.3842e-07, -4.7684e-07]) Copy --- Captum (“comprehension” in Latin) is an open source, extensible library for model interpretability built on PyTorch. With the increase in model complexity and the resulting lack of transparency, model interpretability methods have become increasingly important. Model understanding is both an active area of research as well as an area of focus for practical applications across industries using machine learning. Captum provides state-of-the-art algorithms, including Integrated Gradients, to provide researchers and developers with an easy way to understand which features are contributing to a model’s output. Captum helps ML researchers more easily implement interpretability algorithms that can interact with PyTorch models. It also allows researchers to quickly benchmark their work against other existing algorithms available in the library. For model developers, Captum can be used to improve and troubleshoot models by facilitating the identification of different features that contribute to a model’s output in order to design better models and troubleshoot unexpected model outputs. Target Audience The primary audiences for Captum are model developers who are looking to improve their models and understand which features are important and interpretability researchers focused on identifying algorithms that can better interpret many types of models. Captum can also be used by application engineers who are using trained models in production. Captum provides easier troubleshooting through improved model interpretability, and the potential for delivering better explanations to end users on why they’re seeing a specific piece of content, such as a movie recommendation. Getting Started → --- The tutorials here will help you understand and use Captum. They assume that you are familiar with PyTorch and its basic features.If you are new to Captum, the easiest way to get started is with the Getting started with Captum tutorial.If you are new to PyTorch, the easiest way to get started is with the What is PyTorch? tutorial.Getting started with Captum:In this tutorial we create and train a simple neural network on the Titanic survival dataset. We then use Integrated Gradients to analyze feature importance. We then deep dive the network to assess layer and neuron importance using conductance. Finally, we analyze a specific neuron to understand feature importance for that specific neuron. Find the tutorial here.AttributionInterpreting text models:In this tutorial we use a pre-trained CNN model for sentiment analysis on an IMDB dataset. We use Captum and Integrated Gradients to interpret model predictions by show which specific words have highest attribution to the model output. Find the tutorial here .Interpreting vision with CIFAR:This tutorial demonstrates how to use Captum for interpreting vision focused models. First we create and train (or use a pre-trained) a simple CNN model on the CIFAR dataset. We then interpret the output of an example with a series of overlays using Integrated Gradients and DeepLIFT. Find the tutorial here.Interpreting vision with Pretrained models:Like the CIFAR based tutorial above, this tutorial demonstrates how to use Captum for interpreting vision-focused models. This tutorial begins with a pretrained resnet18 and VGG16 model and demonstrates how to use Intergrated Gradients along with Noise Tunnel, GradientShap, Occlusion, and LRP. Find the tutorial here.Feature ablation on images:This tutorial demonstrates feature ablation in Captum, applied on images as an example. It leverages segmentation masks to define ablation groups over the input features. We show how this kind of analysis helps understanding which parts of the input impacts a certain target in the model. Find the tutorial here.Interpreting multimodal models:To demonstrate interpreting multimodal models we have chosen to look at an open source Visual Question Answer (VQA) model. Using Captum and Integrated Gradients we interpret the output of several test questions and analyze the attribution scores of the text and visual parts of the model. Find the tutorial here.Understanding Llama2 with Captum LLM Attribution:This tutorial demonstrates how to easily use the LLM attribution functionality to interpret the large langague models (LLM) in text generation. It takes Llama2 as the example and shows the step-by-step improvements from the basic attribution setting to more advanced techniques. Find the tutorial here.Interpreting question answering with BERT Part 1:This tutorial demonstrates how to use Captum to interpret a BERT model for question answering. We use a pre-trained model from Hugging Face fine-tuned on the SQUAD dataset and show how to use hooks to examine and better understand embeddings, sub-embeddings, BERT, and attention layers. Find the tutorial here.Interpreting question answering with BERT Part 2:In the second part of Bert tutorial we analyze attention matrices using attribution algorithms s.a. Integrated Gradients. This analysis helps us to identify strong interaction pairs between different tokens for a specific model prediction. We compare our findings with the vector norms and show that attribution scores are more meaningful compared to the vector norms. Find the tutorial here.Interpreting a regression model of California house prices:To demonstrate interpreting regression models we have chosen to look at the California house prices dataset. Using Captum and a variety of attribution methods, we evaluate feature importance as well as internal attribution to understand the network function. Find the tutorial here.Interpreting a semantic segmentation model:In this tutorial, we demonstrate applying Captum to a semantic segmentation task to understand what pixels and regions contribute to the labeling of a particular class. We explore applying GradCAM as well as Feature Ablation to a pretrained Fully-Convolutional Network model with a ResNet-101 backbone. Find the tutorial here.Using Captum with torch.distributed:This tutorial provides examples of using Captum with the torch.distributed package and DataParallel, allowing attributions to be computed in a distributed manner across processors, machines or GPUs. Find the tutorial here.Interpreting DLRM models with Captum:This tutorial demonstrates how we use Captum for Deep Learning Recommender Models usingdlrm model published from facebook research and integrated gradients algorithm. It showcases feature importance differences for sparse and dense features in predicting clicked and non-clicked Ads. It also analyzes the importance of feature interaction layer and neuron importances in the final fully connected layer when predicting clicked Ads. Find the tutorial here.Interpreting vision and text models with LIME:This tutorial demonstrates how to interpret computer vision and text classification models using Local Interpretable Model-agnostic Explanations (LIME) algorithm. For vision it uses resnet18 model to explain image classification based on super-pixels extracted by a segmentation mask. For text it uses a classification model trained on `AG_NEWS` dataset and explains model predictions based on the word tokens in the input text. Find the tutorial here.RobustnessApplying robustness attacks and metrics to CIFAR model and dataset:This tutorial demonstrates how to apply robustness attacks such as FGSM and PGD as well as robustness metrics such as MinParamPerturbation and AttackComparator to a model trained on CIFAR dataset. Apart from that it also demonstrates how robustness techniques can be used in conjunction with attribution algorithms. Find the tutorial here.ConceptTCAV for image classification for googlenet model:This tutorial demonstrates how to apply Testing with Concept Activation Vectors (TCAV) algorithm on image classification problem. It uses googlenet model and imagenet images to showcase the effectiveness of TCAV algorithm on interpreting zebra predictions through stripes concepts. Find the tutorial here.TCAV for NLP sentiment analysis model:This tutorial demonstrates how to apply TCAV algorithm for a NLP task using movie rating dataset and a CNN-based binary sentiment classification model. It showcases that `positive adjectives` concept plays a significant role in predicting positive sentiment. Find the tutorial here.Influential ExamplesIdentifying influential examples and mis-labelled examples with TracInCP:This tutorial demonstrates two use cases of the TracInCP method: providing interpretability by identifying influential training examples for a given prediction, and identifying mis-labelled examples. These two use cases are demonstrated using the CIFAR dataset and checkpoints obtained from training a simple CNN model on it (which can also be downloaded to avoid training). Find the tutorial here. --- Captum API Reference¶ API Reference Attribution Integrated Gradients Saliency DeepLift DeepLiftShap GradientShap Input X Gradient Guided Backprop Guided GradCAM Deconvolution Feature Ablation Occlusion Feature Permutation Shapley Value Sampling Lime KernelShap LRP LLM Attribution Classes LLMAttribution LLMGradientAttribution LLMAttributionResult NoiseTunnel NoiseTunnel Layer Attribution Layer Conductance Layer Activation Internal Influence Layer Gradient X Activation GradCAM Layer DeepLift Layer DeepLiftShap Layer GradientShap Layer Integrated Gradients Layer Feature Ablation Layer Feature Permutation Layer LRP Neuron Attribution Neuron Gradient Neuron Integrated Gradients Neuron Conductance Neuron DeepLift Neuron DeepLiftShap Neuron GradientShap Neuron Guided Backprop Neuron Deconvolution Neuron Feature Ablation Metrics Infidelity Sensitivity Robustness FGSM PGD Attack Comparator Min Param Perturbation Concept-based Interpretability TCAV ConceptInterpreter Concept Classifier Influential Examples DataInfluence SimilarityInfluence TracInCPBase TracInCP TracInCPFast TracInCPFastRandProj Module BinaryConcreteStochasticGates GaussianStochasticGates Utilities Interpretable Input Visualization Interpretable Embeddings Token Reference Base Linear Models Baselines Base Classes Attribution Layer Attribution Neuron Attribution Gradient Attribution Perturbation Attribution Indices and Tables¶ Index Module Index Search Page Captum Navigation API Reference Attribution LLM Attribution Classes NoiseTunnel Layer Attribution Neuron Attribution Metrics Robustness Concept-based Interpretability Influential Examples Module Utilities Base Classes Related Topics Documentation overview Next: Attribution