Visualizing and Interpreting Convolutional Neural Network

Papers

Deconvolutional Networks

• paper: http://www.matthewzeiler.com/pubs/cvpr2010/cvpr2010.pdf
• video: https://ipam.wistia.com/medias/zd0qnekkwc
• presentation: https://mathinstitutes.org/videos/videos/3295

Visualizing and Understanding Convolutional Network

• intro: ECCV 2014
• arxiv: http://arxiv.org/abs/1311.2901
• slides: https://courses.cs.washington.edu/courses/cse590v/14au/cse590v_dec5_DeepVis.pdf
• slides: http://videolectures.net/site/normal_dl/tag=921098/eccv2014_zeiler_convolutional_networks_01.pdf
• video: http://videolectures.net/eccv2014_zeiler_convolutional_networks/
• chs: http://blog.csdn.net/kklots/article/details/17136059
• github: https://github.com/piergiaj/caffe-deconvnet

Deep Inside Convolutional Networks: Visualising Image Classification Models and Saliency Maps

• intro: ICLR 2014 workshop
• arxiv: http://arxiv.org/abs/1312.6034
• github: https://github.com/yasunorikudo/vis-cnn

Understanding Deep Image Representations by Inverting Them

• arxiv: http://arxiv.org/abs/1412.0035
• github: https://github.com/aravindhm/deep-goggle

deepViz: Visualizing Convolutional Neural Networks for Image Classification

• paper: http://vis.berkeley.edu/courses/cs294-10-fa13/wiki/images/f/fd/DeepVizPaper.pdf
• github: https://github.com/bruckner/deepViz

Inverting Convolutional Networks with Convolutional Networks

• arxiv: http://arxiv.org/abs/1506.02753

Understanding Neural Networks Through Deep Visualization

• project page: http://yosinski.com/deepvis
• arxiv: http://arxiv.org/abs/1506.06579
• github: https://github.com/yosinski/deep-visualization-toolbox

Visualizing Higher-Layer Features of a Deep Network

• paper: http://www.iro.umontreal.ca/~lisa/publications2/index.php/publications/show/247

Generative Modeling of Convolutional Neural Networks

• project page: http://www.stat.ucla.edu/~yang.lu/Project/generativeCNN/main.html
• arxiv: http://arxiv.org/abs/1412.6296
• code: http://www.stat.ucla.edu/~yang.lu/Project/generativeCNN/doc/caffe-generative.zip

Understanding Intra-Class Knowledge Inside CNN

• arxiv: http://arxiv.org/abs/1507.02379

Learning FRAME Models Using CNN Filters for Knowledge Visualization

• project page: http://www.stat.ucla.edu/~yang.lu/project/deepFrame/main.html
• arxiv: http://arxiv.org/abs/1509.08379
• code: http://www.stat.ucla.edu/~yang.lu/project/deepFrame/doc/code.zip

Convergent Learning: Do different neural networks learn the same representations?

• intro: ICLR 2016
• arxiv: http://arxiv.org/abs/1511.07543
• github: https://github.com/yixuanli/convergent_learning
• video: http://videolectures.net/iclr2016_yosinski_convergent_learning/

Visualizing and Understanding Deep Texture Representations

• homepage: http://vis-www.cs.umass.edu/texture/
• arxiv: http://arxiv.org/abs/1511.05197
• paper: https://people.cs.umass.edu/~smaji/papers/texture-cvpr16.pdf

Visualizing Deep Convolutional Neural Networks Using Natural Pre-Images

• arxiv: http://arxiv.org/abs/1512.02017

An Interactive Node-Link Visualization of Convolutional Neural Networks

• homepage: http://scs.ryerson.ca/~aharley/vis/
• code: http://scs.ryerson.ca/~aharley/vis/source.zip
• demo: http://scs.ryerson.ca/~aharley/vis/conv/
• review: http://www.popsci.com/gaze-inside-mind-artificial-intelligence

Learning Deep Features for Discriminative Localization

• project page: http://cnnlocalization.csail.mit.edu/
• arxiv: http://arxiv.org/abs/1512.04150
• github: https://github.com/metalbubble/CAM
• blog: http://jacobcv.blogspot.com/2016/08/class-activation-maps-in-keras.html
• github: https://github.com/jacobgil/keras-cam

Multifaceted Feature Visualization: Uncovering the Different Types of Features Learned By Each Neuron in Deep Neural Networks

• intro: Visualization for Deep Learning workshop. ICML 2016
• arxiv: http://arxiv.org/abs/1602.03616
• homepage: http://www.evolvingai.org/nguyen-yosinski-clune-2016-multifaceted-feature
• github: https://github.com/Evolving-AI-Lab/mfv

A New Method to Visualize Deep Neural Networks

• arxiv: http://arxiv.org/abs/1603.02518

A Taxonomy and Library for Visualizing Learned Features in Convolutional Neural Networks

• arxiv: http://arxiv.org/abs/1606.07757

VisualBackProp: visualizing CNNs for autonomous driving

VisualBackProp: efficient visualization of CNNs

• arxiv: https://arxiv.org/abs/1611.05418
• github: https://github.com/mbojarski/VisualBackProp

Grad-CAM: Why did you say that? Visual Explanations from Deep Networks via Gradient-based Localization

Grad-CAM: Visual Explanations from Deep Networks via Gradient-based Localization

• arxiv: https://arxiv.org/abs/1610.02391
• github: https://github.com/ramprs/grad-cam/
• github(Keras): https://github.com/jacobgil/keras-grad-cam
• github(TensorFlow): https://github.com/Ankush96/grad-cam.tensorflow

Grad-CAM: Why did you say that?

• intro: NIPS 2016 Workshop on Interpretable Machine Learning in Complex Systems
• intro: extended abstract version of arXiv:1610.02391
• arxiv: https://arxiv.org/abs/1611.07450

Visualizing Residual Networks

• intro: UC Berkeley CS 280 final project report
• arxiv: https://arxiv.org/abs/1701.02362

Visualizing Deep Neural Network Decisions: Prediction Difference Analysis

• intro: University of Amsterdam & Canadian Institute of Advanced Research & Vrije Universiteit Brussel
• intro: ICLR 2017
• arxiv: https://arxiv.org/abs/1702.04595
• github: https://github.com/lmzintgraf/DeepVis-PredDiff

ActiVis: Visual Exploration of Industry-Scale Deep Neural Network Models

• intro: Georgia Tech & Facebook
• arxiv: https://arxiv.org/abs/1704.01942

Picasso: A Neural Network Visualizer

• arxiv: https://arxiv.org/abs/1705.05627
• github: https://github.com/merantix/picasso
• blog: https://medium.com/merantix/picasso-a-free-open-source-visualizer-for-cnns-d8ed3a35cfc5

CNN Fixations: An unraveling approach to visualize the discriminative image regions

• arxiv: https://arxiv.org/abs/1708.06670
• github: https://github.com/utsavgarg/cnn-fixations

A Forward-Backward Approach for Visualizing Information Flow in Deep Networks

• intro: NIPS 2017 Symposium on Interpretable Machine Learning. Iowa State University
• arxiv: https://arxiv.org/abs/1711.06221

Using KL-divergence to focus Deep Visual Explanation

https://arxiv.org/abs/1711.06431

An Introduction to Deep Visual Explanation

• intro: NIPS 2017 - Workshop Interpreting, Explaining and Visualizing Deep Learning
• arxiv: https://arxiv.org/abs/1711.09482

Visual Explanation by Interpretation: Improving Visual Feedback Capabilities of Deep Neural Networks

https://arxiv.org/abs/1712.06302

Visualizing the Loss Landscape of Neural Nets

• intro: University of Maryland & United States Naval Academy
• arxiv: https://arxiv.org/abs/1712.09913

Visualizing Deep Similarity Networks

https://arxiv.org/abs/1901.00536

Interpreting Convolutional Neural Networks

Network Dissection: Quantifying Interpretability of Deep Visual Representations

• intro: CVPR 2017 oral. MIT
• project page: http://netdissect.csail.mit.edu/
• arxiv: https://arxiv.org/abs/1704.05796
• github: https://github.com/CSAILVision/NetDissect

Interpreting Deep Visual Representations via Network Dissection

https://arxiv.org/abs/1711.05611

Methods for Interpreting and Understanding Deep Neural Networks

• intro: Technische Universit¨at Berlin & Fraunhofer Heinrich Hertz Institute
• arxiv: https://arxiv.org/abs/1706.07979

SVCCA: Singular Vector Canonical Correlation Analysis for Deep Learning Dynamics and Interpretability

• intro: NIPS 2017. Google Brain & Uber AI Labs
• arxiv: https://arxiv.org/abs/1706.05806
• github: https://github.com/google/svcca/
• blog: https://research.googleblog.com/2017/11/interpreting-deep-neural-networks-with.html

Towards Interpretable Deep Neural Networks by Leveraging Adversarial Examples

• intro: Tsinghua University
• arxiv: https://arxiv.org/abs/1708.05493

Interpretable Convolutional Neural Networks

https://arxiv.org/abs/1710.00935

Interpreting Convolutional Neural Networks Through Compression

• intro: NIPS 2017 Symposium on Interpretable Machine Learning
• arxiv: https://arxiv.org/abs/1711.02329

Interpreting Deep Neural Networks

• blog: http://www.shallowmind.co/jekyll/pixyll/2017/12/30/tree-regularization/

Interpreting CNNs via Decision Trees

https://arxiv.org/abs/1802.00121

Visual Interpretability for Deep Learning: a Survey

https://arxiv.org/abs/1802.00614

Interpreting Deep Classifier by Visual Distillation of Dark Knowledge

• intro: University of Edinburgh & Huawei Research America
• arxiv: https://arxiv.org/abs/1803.04042

How convolutional neural network see the world - A survey of convolutional neural network visualization methods

• intro: Mathematical Foundations of Computing. George Mason University & Clarkson University
• arxiv: https://arxiv.org/abs/1804.11191

Understanding Regularization to Visualize Convolutional Neural Networks

• intro: Konica Minolta Laboratory Europe & Technical University of Munich
• arxiv: https://arxiv.org/abs/1805.00071

Deeper Interpretability of Deep Networks

• intro: University of Glasgow & University of Oxford & University of California
• arxiv: https://arxiv.org/abs/1811.07807

Interpretable CNNs

https://arxiv.org/abs/1901.02413

Explaining AlphaGo: Interpreting Contextual Effects in Neural Networks

https://arxiv.org/abs/1901.02184

Interpretable BoW Networks for Adversarial Example Detection

https://arxiv.org/abs/1901.02229

Deep Features Analysis with Attention Networks

• intro: In AAAI-19 Workshop on Network Interpretability for Deep Learning
• arxiv: https://arxiv.org/abs/1901.10042

Understanding Neural Networks via Feature Visualization: A survey

• intro: A book chapter in an Interpretable ML book (http://www.interpretable-ml.org/book/)
• arxiv: https://arxiv.org/abs/1904.08939

Explaining Neural Networks via Perturbing Important Learned Features

https://arxiv.org/abs/1911.11081

Interpreting Adversarially Trained Convolutional Neural Networks

• intro: ICML 2019
• arxiv: https://arxiv.org/abs/1905.09797
• github: https://github.com/PKUAI26/AT-CNN

Projects

Interactive Deep Neural Net Hallucinations

• project page: http://317070.github.io/Dream/
• github: https://github.com/317070/Twitch-plays-LSD-neural-net

torch-visbox

• github: https://github.com/Aysegul/torch-visbox

draw_convnet: Python script for illustrating Convolutional Neural Network (ConvNet)

• github: https://github.com/gwding/draw_convnet

Caffe prototxt visualization

• intro: Recommended by Kaiming He
• github: https://github.com/ethereon/netscope
• quickstart: http://ethereon.github.io/netscope/quickstart.html
• demo: http://ethereon.github.io/netscope/#/editor

Keras Visualization Toolkit

• github: https://github.com/raghakot/keras-vis
• docs: https://raghakot.github.io/keras-vis/

mNeuron: A Matlab Plugin to Visualize Neurons from Deep Models

• project page: http://vision03.csail.mit.edu/cnn_art/
• github: https://github.com/donglaiw/mNeuron

cnnvis-pytorch

• intro: visualization of CNN in PyTorch
• github: https://github.com/leelabcnbc/cnnvis-pytorch

VisualDL

• intro: A platform to visualize the deep learning process
• homepage: http://visualdl.paddlepaddle.org/
• github: https://github.com/PaddlePaddle/VisualDL

Blogs

“Visualizing GoogLeNet Classes”

http://auduno.com/post/125362849838/visualizing-googlenet-classes

Visualizing CNN architectures side by side with mxnet

• blog: http://josephpcohen.com/w/visualizing-cnn-architectures-side-by-side-with-mxnet/

How convolutional neural networks see the world: An exploration of convnet filters with Keras

• blog: http://blog.keras.io/how-convolutional-neural-networks-see-the-world.html
• github: https://github.com/fchollet/keras/blob/master/examples/conv_filter_visualization.py

Visualizing Deep Learning with t-SNE (Tutorial and Video)

• blog: https://medium.com/@awjuliani/visualizing-deep-learning-with-t-sne-tutorial-and-video-e7c59ee4080c#.ubhijafw7
• github: https://github.com/awjuliani/3D-TSNE

Peeking inside Convnets

• blog: https://auduno.github.io/2016/06/18/peeking-inside-convnets/

Visualizing Features from a Convolutional Neural Network

• blog: http://kvfrans.com/visualizing-features-from-a-convolutional-neural-network/
• github: https://github.com/kvfrans/feature-visualization

Visualizing Deep Neural Networks Classes and Features

http://ankivil.com/visualizing-deep-neural-networks-classes-and-features/

Visualizing parts of Convolutional Neural Networks using Keras and Cats

• blog: https://hackernoon.com/visualizing-parts-of-convolutional-neural-networks-using-keras-and-cats-5cc01b214e59#.bt6bb13dk
• github: https://github.com/erikreppel/visualizing_cnns

Visualizing convolutional neural networks

• intro: How to build convolutional neural networks from scratch w/ Tensorflow
• blog: https://www.oreilly.com/ideas/visualizing-convolutional-neural-networks
• github: https://github.com//wagonhelm/Visualizing-Convnets/

Tools

Topological Visualisation of a Convolutional Neural Network

http://terencebroad.com/convnetvis/vis.html

Visualization of Places-CNN and ImageNet CNN

• homepage: http://places.csail.mit.edu/visualizationCNN.html
• DrawCNN: http://people.csail.mit.edu/torralba/research/drawCNN/drawNet.html

Visualization of a feed forward Neural Network using MNIST dataset

• homepage: http://nn-mnist.sennabaum.com/
• github: https://github.com/csenn/nn-visualizer

CNNVis: Towards Better Analysis of Deep Convolutional Neural Networks.

http://shixialiu.com/publications/cnnvis/demo/

Quiver: Interactive convnet features visualization for Keras

• homepage: https://jakebian.github.io/quiver/
• github: https://github.com/jakebian/quiver

Netron

• intro: Visualizer for deep learning and machine learning models
• github: https://github.com/lutzroeder/netron