Papers

Accelerating Deep Convolutional Neural Networks Using Specialized Hardware

• paper: http://research.microsoft.com/pubs/240715/CNN%20Whitepaper.pdf

Installation / Deploying

Setting up a Deep Learning Machine from Scratch (Software): Instructions for setting up the software on your deep learning machine

• intro: A detailed guide to setting up your machine for deep learning research. Includes instructions to install drivers, tools and various deep learning frameworks. This was tested on a 64 bit machine with Nvidia Titan X, running Ubuntu 14.04
• github: https://github.com/saiprashanths/dl-setup

How to install CUDA Toolkit and cuDNN for deep learning

• blog: http://www.pyimagesearch.com/2016/07/04/how-to-install-cuda-toolkit-and-cudnn-for-deep-learning/

Deploying Deep Learning: Guide to deploying deep-learning inference networks and realtime object detection with TensorRT and Jetson TX1.

• github: https://github.com/dusty-nv/jetson-inference

Install Log

• intro: setting up Caffe on a cluster running Redhat 6.3 (Santiago) without having root
• github: https://github.com/yosinski/caffe/blob/jason_public/doc/linux-no-root-install-log.md

Lessons Learned from Deploying Deep Learning at Scale

• blog: http://blog.algorithmia.com/deploying-deep-learning-cloud-services/

Docker

All-in-one Docker image for Deep Learning

• intro: An all-in-one Docker image for deep learning. Contains all the popular DL frameworks (TensorFlow, Theano, Torch, Caffe, etc.)
• github: https://github.com/saiprashanths/dl-docker

NVIDIA Docker: GPU Server Application Deployment Made Easy

• blog: https://devblogs.nvidia.com/parallelforall/nvidia-docker-gpu-server-application-deployment-made-easy/
• github: https://github.com/NVIDIA/nvidia-docker

Deep learning base image for Docker (Tensorflow, Caffe, MXNet, Torch, Openface, etc.)

https://github.com/dominiek/deep-base

Deepo: a Docker image with a full reproducible deep learning research environment

• intro: A Docker image containing almost all popular deep learning frameworks: theano, tensorflow, sonnet, pytorch, keras, lasagne, mxnet, cntk, chainer, caffe, torch.
• project page: https://hub.docker.com/r/ufoym/deepo/
• github: https://github.com//ufoym/deepo

Cloud

SuperVessel Cloud for POWER/OpenPOWER LoginRegisterTutorials

http://www.ptopenlab.com/

Building Deep Neural Networks in the Cloud with Azure GPU VMs, MXNet and Microsoft R Server

https://blogs.technet.microsoft.com/machinelearning/2016/09/15/building-deep-neural-networks-in-the-cloud-with-azure-gpu-vms-mxnet-and-microsoft-r-server/

Microsoft open sources its next-gen cloud hardware design

• blog: https://techcrunch.com/2016/10/31/microsoft-open-sources-its-next-gen-cloud-hardware-design/

Google Taps AMD For Accelerating Machine Learning In The Cloud

http://www.forbes.com/sites/aarontilley/2016/11/15/google-taps-amd-for-accelerating-machine-learning-in-the-cloud/#3549d8554181

Amazon EC2

Deep Learning AMI on AWS Marketplace

https://aws.amazon.com/marketplace/pp/B01M0AXXQB

We Have To Go Deeper: AWS p2.xlarge GPU optimized deep learning cluster-grenade

• github: https://github.com/Miej/GoDeeper

A GPU enabled AMI for Deep Learning

• blog: https://blog.empiricalci.com/a-gpu-enabled-ami-for-deep-learning-5aa3d694b630#.9339zxm4e

Keras with GPU on Amazon EC2 – a step-by-step instruction

https://medium.com/@mateuszsieniawski/keras-with-gpu-on-amazon-ec2-a-step-by-step-instruction-4f90364e49ac#.k27d0mqir

Microsoft R Server

Training Deep Neural Networks on ImageNet Using Microsoft R Server and Azure GPU VMs

• blog: https://blogs.technet.microsoft.com/machinelearning/2016/11/15/imagenet-deep-neural-network-training-using-microsoft-r-server-and-azure-gpu-vms/

Hardware System

I: Building a Deep Learning (Dream) Machine

• blog: http://graphific.github.io/posts/building-a-deep-learning-dream-machine/
• slides: http://www.slideshare.net/roelofp/building-a-deep-learning-dream-machine

II: Running a Deep Learning (Dream) Machine

• blog: http://graphific.github.io/posts/running-a-deep-learning-dream-machine/

A Full Hardware Guide to Deep Learning

• blog: http://timdettmers.com/2015/03/09/deep-learning-hardware-guide/

Build your own Deep Learning Box

• blog: https://annalyzin.wordpress.com/2016/05/19/build-a-deep-learning-box/

32-TFLOP Deep Learning GPU Box: A super-fast linux-based machine with multiple GPUs for training deep neural nets

https://hackaday.io/project/12070-32-tflop-deep-learning-gpu-box

Hands-on with the NVIDIA DIGITS DevBox for Deep Learning

• blog: http://www.pyimagesearch.com/2016/06/06/hands-on-with-the-nvidia-digits-devbox-for-deep-learning/

Considerations when setting up deep learning hardware

• blog: http://www.pyimagesearch.com/2016/06/13/considerations-when-setting-up-deep-learning-hardware/

Building a Workstation for Deep Learning

• slides: http://www.slideshare.net/PetteriTeikariPhD/deep-learning-workstation

Deep Learning Machine: First build experience

• blog: https://medium.com/@vivek.yadav/deep-learning-machine-first-build-experience-d04abf198831#.1d6q5mw9m

Building a machine learning/deep learning workstation for under $5000

• blog: https://www.analyticsvidhya.com/blog/2016/11/building-a-machine-learning-deep-learning-workstation-for-under-5000/

Hardware Guide: Neural Networks on GPUs (Updated 2016-1-30)

• intro: by Joseph Redmon
• blog: http://pjreddie.com/darknet/hardware-guide/

Building Your Own Deep Learning Box

https://medium.com/@bfortuner/building-your-own-deep-learning-box-47b918aea1eb#.4r5zchk4f

Setting up a Deep learning machine in a lazy yet quick way https://medium.com/@sravsatuluri/setting-up-a-deep-learning-machine-in-a-lazy-yet-quick-way-be2642318850#.jrxrkfxa2

Deep Confusion: Misadventures In Building A Deep Learning Machine

http://www.topbots.com/deep-confusion-misadventures-in-building-a-machine-learning-server/

DIY-Deep-Learning-Workstation

• intro: Build a deep learning workstation from scratch (HW & SW).
• github: https://github.com/charlesq34/DIY-Deep-Learning-Workstation

GPU

Which GPU(s) to Get for Deep Learning: My Experience and Advice for Using GPUs in Deep Learning

• blog: http://timdettmers.com/2017/04/09/which-gpu-for-deep-learning/

从深度学习选择什么样的gpu来谈谈gpu的硬件架构

• blog: http://chenrudan.github.io/blog/2015/12/20/introductionofgpuhardware.html

GPU折腾手记——2015 (by 李沐)

• blog: http://mli.github.io/gpu/2016/01/17/build-gpu-clusters/

HPC, Deep Learning and GPUs(2016 Stanford HPC Conference)

• youtube: https://www.youtube.com/watch?v=JwgoC-1V_38
• video: http://pan.baidu.com/s/1pKrSvOZ

Modern GPU 2.0: Design patterns for GPU computing

• intro: Modern GPU is code and commentary intended to promote new and productive ways of thinking about GPU computing.
• homepage: http://nvlabs.github.io/moderngpu/
• github: https://github.com/nvlabs/moderngpu

CuMF: CUDA-Acclerated ALS on mulitple GPUs.

• github: https://github.com/wei-tan/CuMF

Basic Performance Analysis of NVIDIA GPU Accelerator Cards for Deep Learning Applications

• wihte paper: https://www.amax.com/enterprise/pdfs/Deep%20Learning%20Performance%20Analysis.pdf

CuPy : NumPy-like API accelerated with CUDA

• github: https://github.com/pfnet/cupy

NumPy GPU acceleration

• blog: http://scottsievert.com/blog/2016/07/01/numpy-gpu/

Efficient Convolutional Neural Network Inference on Mobile GPUs (Embedded Vision Summit)

• youtube: https://www.youtube.com/watch?v=ximyhmm17UM

Deep Learning with Multiple GPUs on Rescale: Torch

• blog: https://blog.rescale.com/deep-learning-with-multiple-gpus-on-rescale-torch/

GPU-accelerated Theano & Keras on Windows 10 native

• arxiv: https://github.com/philferriere/dlwin

NVIDIA Announces Quadro GP100 - Big Pascal Comes to Workstations

http://www.anandtech.com/show/11102/nvidia-announces-quadro-gp100

FPGA

Recurrent Neural Networks Hardware Implementation on FPGA

• arxiv: http://arxiv.org/abs/1511.05552

Is implementing deep learning on FPGAs a natural next step after the success with GPUs?

• quora: https://www.quora.com/Is-implementing-deep-learning-on-FPGAs-a-natural-next-step-after-the-success-with-GPUs

Efficient Implementation of Neural Network Systems Built on FPGAs, Programmed with OpenCL

• paper: https://www.altera.com/content/dam/altera-www/global/en_US/pdfs/literature/solution-sheets/efficient_neural_networks.pdf?utm_source=Altera&utm_medium=link&utm_campaign=OpenCL_15_1&utm_content=NA_efficient-neural-networks-solution-sheet-download-link

Deep Learning on FPGAs: Past, Present, and Future

• arxiv: http://arxiv.org/abs/1602.04283

FPGAs Challenge GPUs as a Platform for Deep Learning

• blog: https://www.tractica.com/automation-robotics/fpgas-challenge-gpus-as-a-platform-for-deep-learning/

Convolution Neural Network CNN Implementation on Altera FPGA using OpenCL

• youtube: https://www.youtube.com/watch?v=78Qd5t-Mn0s

Accelerating Deep Learning Using Altera FPGAs (Embedded Vision Summit)

• youtube: https://www.youtube.com/watch?v=HlBC9qBqZRs
• slides: http://www.slideshare.net/embeddedvision/accelerating-deep-learning-using-altera-fpgas-a-presentation-from-intel

Machine Learning on FPGAs: Neural Networks

• youtube: https://www.youtube.com/watch?v=3iCifD8gZ0Q

Comprehensive Evaluation of OpenCL-based Convolutional Neural Network Accelerators in Xilinx and Altera FPGAs

• arxiv: https://arxiv.org/abs/1609.09296

Microsoft Goes All in for FPGAs to Build Out AI Cloud

• blog: https://www.top500.org/news/microsoft-goes-all-in-for-fpgas-to-build-out-cloud-based-ai/

Caffeinated FPGAs: FPGA Framework For Convolutional Neural Networks

• arxiv: https://arxiv.org/abs/1609.09671
• github: https://github.com/dicecco1/fpga_caffe

Intel Unveils FPGA to Accelerate Neural Networks

http://datacenterfrontier.com/intel-unveils-fpga-to-accelerate-ai-neural-networks/

Deep Learning with FPGA

• blog: https://amundtveit.com/2016/11/23/deep-learning-with-fpga/

A General Neural Network Hardware Architecture on FPGA

• intro: University of Birmingham
• arxiv: https://arxiv.org/abs/1711.05860

Approximate FPGA-based LSTMs under Computation Time Constraints

• intro: ARC 2018
• arxiv: https://arxiv.org/abs/1801.02190

ARM / Processor

‘Neural network’ spotted deep inside Samsung’s Galaxy S7 silicon brain: Secrets of Exynos M1 cores spilled

• blog: http://www.theregister.co.uk/2016/08/22/samsung_m1_core/?mt=1471918256061

Intel will add deep-learning instructions to its processors

• blog: http://lemire.me/blog/2016/10/14/intel-will-add-deep-learning-instructions-to-its-processors/

SRAM

ShiDianNao: Shifting Vision Processing Closer to the Sensor http://lap.epfl.ch/files/content/sites/lap/files/shared/publications/DuJun15_ShiDianNaoShiftingVisionProcessingCloserToTheSensor_ISCA15.pdf

Blogs

Emerging “Universal” FPGA, GPU Platform for Deep Learning

• blog: http://www.nextplatform.com/2016/06/29/universal-fpga-gpu-platform-deep-learning/

An Early Look at Startup Graphcore’s Deep Learning Chip

https://www.nextplatform.com/2017/03/09/early-look-startup-graphcores-deep-learning-chip/

Hardware for Deep Learning

https://medium.com/towards-data-science/hardware-for-deep-learning-8d9b03df41a

Videos

Energy-efficient Hardware for Embedded Vision and Deep Convolutional Neural Networks

• intro: September 2016 Embedded Vision Alliance Member Meeting Presentation: MIT
• youtube: https://www.youtube.com/watch?v=dO_lHz87DVM

ImageNet

Amazon DSSTNE

Amazon DSSTNE: Deep Scalable Sparse Tensor Network Engine

• intro: Deep Scalable Sparse Tensor Network Engine (DSSTNE) is an Amazon developed library for building Deep Learning (DL) machine learning (ML) models
• github: https://github.com/amznlabs/amazon-dsstne

Apache SINGA

• project-website: http://singa.incubator.apache.org/
• github: https://github.com/apache/incubator-singa
• paper: http://www.comp.nus.edu.sg/~ooibc/singaopen-mm15.pdf
• paper: http://www.comp.nus.edu.sg/~ooibc/singa-tomm.pdf

Blocks

Blocks: A Theano framework for building and training neural networks

• github: https://github.com/mila-udem/blocks

Blocks and Fuel: Frameworks for deep learning

• arxiv: http://arxiv.org/abs/1506.00619

BrainCore

BrainCore: The iOS and OS X neural network framework

https://github.com/aleph7/BrainCore

Brainstorm

Brainstorm: Fast, flexible and fun neural networks

• github: https://github.com/IDSIA/brainstorm

Caffe

Caffe: Convolutional Architecture for Fast Feature Embedding

• github: https://github.com/BVLC/caffe
• paper: http://arxiv.org/abs/1408.5093
• tutorial: http://tutorial.caffe.berkeleyvision.org/
• slides: http://vision.stanford.edu/teaching/cs231n/slides/caffe_tutorial.pdf
• slides: http://vision.princeton.edu/courses/COS598/2015sp/slides/Caffe/caffe_tutorial.pdf
• caffe-doc: http://caffe.berkeleyvision.org/doxygen/index.html
• tutorials(“CAFFE with CUDA”): http://pan.baidu.com/s/1i4kmpyH

OpenCL Caffe

• intro: an experimental, community-maintained branch
• github: https://github.com/BVLC/caffe/tree/opencl

Caffe on both Linux and Windows

• github: https://github.com/Microsoft/caffe

ApolloCaffe: a fork of Caffe that supports dynamic networks

• homepage: http://apollocaffe.com/
• github: http://github.com/Russell91/apollocaffe

fb-caffe-exts: Some handy utility libraries and tools for the Caffe deep learning framework

• intro: fb-caffe-exts is a collection of extensions developed at FB while using Caffe in (mainly) production scenarios.
• github: https://github.com/facebook/fb-caffe-exts

Caffe-Android-Lib: Porting caffe to android platform

• github: https://github.com/sh1r0/caffe-android-lib

caffe-android-demo: An android caffe demo app exploiting caffe pre-trained ImageNet model for image classification

• github: https://github.com/sh1r0/caffe-android-demo

Caffe.js: Run Caffe models in the browser using ConvNetJS

• github: https://github.com/chaosmail/caffejs/
• demo: http://chaosmail.github.io/caffejs/models.html

Intel Caffe

• intro: This fork of BVLC/Caffe is dedicated to improving performance of this deep learning framework when running on CPU, in particular Intel® Xeon processors (HSW+) and Intel® Xeon Phi processors
• github https://github.com/intel/caffe

NVIDIA Caffe

https://github.com/NVIDIA/caffe

Mini-Caffe

• intro: Minimal runtime core of Caffe, Forward only, GPU support and Memory efficiency.
• github: https://github.com/luoyetx/mini-caffe

Caffe on Mobile Devices

• intro: Optimized (for size and speed) Caffe lib for iOS and Android with demo APP.
• github: https://github.com/solrex/caffe-mobile

CaffeOnACL

• intro: Using ARM Compute Library (NEON+GPU) to speed up caffe; Providing utilities to debug, profile and tune application performance
• github: https://github.com/OAID/caffeOnACL

Multi-GPU / MPI Caffe

Caffe with OpenMPI-based Multi-GPU support

• intro: A fork of Caffe with OpenMPI-based Multi-GPU (mainly data parallel) support for action recognition and more.
• github: https://github.com/yjxiong/caffe/tree/mem

mpi-caffe: Model-distributed Deep Learning with Caffe and MPI

• project page: https://computing.ece.vt.edu/~steflee/mpi-caffe.html
• github: https://github.com/steflee/mpi-caffe

Caffe-MPI for Deep Learning

• github: https://github.com/Caffe-MPI/Caffe-MPI.github.io
• slides: http://mug.mvapich.cse.ohio-state.edu/static/media/mug/presentations/2016/Caffe-MPI_A_Parallel_Framework_on_the_GPU_Clusters.pdf

Caffe Utils

Caffe-model

• intro: Python script to generate prototxt on Caffe, specially the inception_v3\inception_v4\inception_resnet\fractalnet
• github: https://github.com/soeaver/caffe-model

Caffe2

Caffe2: A New Lightweight, Modular, and Scalable Deep Learning Framework

• intro: Caffe2 is a deep learning framework made with expression, speed, and modularity in mind. It is an experimental refactoring of Caffe, and allows a more flexible way to organize computation.
• homepage: https://caffe2.ai/
• github https://github.com/caffe2/caffe2
• github https://github.com/Yangqing/caffe2
• model zoo: https://caffe2.ai/docs/zoo.html
• models: https://github.com/caffe2/models

CDNN2

CDNN2 - CEVA Deep Neural Network Software Framework

• intro: Accelerating the development of Artificial Intelligence and its deployment in Low-Power Embedded Systems

• homepage: http://launch.ceva-dsp.com/cdnn2/

• blog: http://www.tomshardware.com/news/ceva-cdnn2-tensorflow-embedded-systems,32158.html

Chainer

Chainer: a neural network framework

• website: http://chainer.org/
• github: https://github.com/pfnet/chainer
• benchmark: http://chainer.readthedocs.org/en/latest/comparison.html

Introduction to Chainer: Neural Networks in Python

• blog: http://multithreaded.stitchfix.com/blog/2015/12/09/intro-to-chainer/
• github: https://github.com/stitchfix/Algorithms-Notebooks

CNTK

CNTK: Computational Network Toolkit

• github: https://github.com/Microsoft/CNTK
• book: http://research.microsoft.com/pubs/226641/CNTKBook-20160121.pdf
• tutorial: http://research.microsoft.com/en-us/um/people/dongyu/CNTK-Tutorial-NIPS2015.pdf

An Introduction to Computational Networks and the Computational Network Toolkit

http://research.microsoft.com/apps/pubs/?id=226641

ConvNetJS

ConvNetJS: Deep Learning in Javascript. Train Convolutional Neural Networks (or ordinary ones) in your browser

• github: https://github.com/karpathy/convnetjs

DeepBeliefSDK

DeepBeliefSDK: The SDK for Jetpac’s iOS, Android, Linux, and OS X Deep Belief image recognition framework

• github: https://github.com/jetpacapp/DeepBeliefSDK
• demo: https://github.com/jetpacapp/Jetpac-Deep-Belief-Demo-App
• demo: https://github.com/jetpacapp/Jetpac-Deep-Belief-Learner-Demo-App

DeepDetect

DeepDetect: Open Source API & Deep Learning Server

• webiste: http://www.deepdetect.com/
• github: https://github.com/beniz/deepdetect

Deeplearning4j (DL4J)

Deeplearning4j: Deep Learning for Java

• homepage: http://deeplearning4j.org/
• github: https://github.com/deeplearning4j/deeplearning4j

Deeplearning4j images for cuda and hadoop.

• github: https://github.com/deeplearning4j/docker

Deeplearning4J Examples

• intro: Deeplearning4j Examples (DL4J, DL4J Spark, DataVec)
• github: https://github.com/deeplearning4j/dl4j-examples

DeepLearningKit

DeepLearningKit: Open Source Deep Learning Framework for Apple’s tvOS, iOS and OS X

• homepage: http://deeplearningkit.org/
• github: https://github.com/DeepLearningKit/DeepLearningKit

Tutorial — Using DeepLearningKit with iOS for iPhone and iPad

https://medium.com/@atveit/tutorial-using-deeplearningkit-with-ios-for-iphone-and-ipad-de727679bae4#.1bvnhxhjo

DeepSpark

DeepSpark: Deeplearning framework running on Spark

• github: https://github.com/deepspark/deepspark
• homepage: http://deepspark.snu.ac.kr/
• arxiv: http://arxiv.org/abs/1602.08191

DIGITS

DIGITS: the Deep Learning GPU Training System

• homepage: https://developer.nvidia.com/digits
• github: https://github.com/NVIDIA/DIGITS

dp

dp: A deep learning library for streamlining research and development using the Torch7 distribution

• github: https://github.com/nicholas-leonard/dp
• manual: https://dp.readthedocs.org/en/latest/
• manual: https://github.com/nicholas-leonard/dp/blob/master/doc/index.md

Dragon

Dragon: A Computation Graph Virtual Machine Based Deep Learning Framework

• arxiv: https://arxiv.org/abs/1707.08265
• github: https://github.com/neopenx/Dragon

DyNet

**DyNet: The Dynamic Neural Network Toolkit **

• paper: https://arxiv.org/abs/1701.03980
• github: https://github.com/clab/dynet

DyNet Benchmarks

• github: https://github.com/neulab/dynet-benchmark

IDLF

IDLF: The Intel® Deep Learning Framework

• website: https://01.org/zh/intel-deep-learning-framework?langredirect=1
• github: https://github.com/01org/idlf

Keras

Keras: Deep Learning library for Theano and TensorFlow

• github: https://github.com/fchollet/keras
• blog: http://blog.keras.io/introducing-keras-10.html
• docs: http://keras.io/getting-started/functional-api-guide/

MarcBS/keras fork

• github: https://github.com/MarcBS/keras

Hera: Train/evaluate a Keras model, get metrics streamed to a dashboard in your browser.

• github: https://github.com/jakebian/hera

Installing Keras for deep learning

• blog: http://www.pyimagesearch.com/2016/07/18/installing-keras-for-deep-learning/

Keras Applications - deep learning models that are made available alongside pre-trained weights

https://keras.io/applications/

Keras resources: Directory of tutorials and open-source code repositories for working with Keras, the Python deep learning library

• github: https://github.com/fchollet/keras-resources

Keras.js: Run trained Keras models in the browser, with GPU support

• homepage: https://transcranial.github.io/keras-js/
• github: https://github.com/transcranial/keras-js

keras2cpp

• intro: This is a bunch of code to port Keras neural network model into pure C++.
• github: https://github.com/pplonski/keras2cpp

keras-cn: Chinese keras documents with more examples, explanations and tips.

• github: https://github.com/MoyanZitto/keras-cn

Kerasify: Small library for running Keras models from a C++ application

https://github.com/moof2k/kerasify

Knet

Knet: Koç University deep learning framework

• intro: Knet (pronounced “kay-net”) is the Koç University deep learning framework implemented in Julia by Deniz Yuret and collaborators.
• github: https://github.com/denizyuret/Knet.jl
• doc: https://knet.readthedocs.org/en/latest/

Lasagne

Lasagne: Lightweight library to build and train neural networks in Theano

• github: https://github.com/Lasagne/Lasagne
• docs: http://lasagne.readthedocs.org/en/latest/

Leaf

Leaf: The Hacker’s Machine Learning Engine

• homepage: http://autumnai.github.io/leaf/leaf/index.html
• github: https://github.com/autumnai/leaf
• homepage: http://autumnai.com/leaf/book/leaf.html
• homepage(“The Hacker’s Machine Intelligence Platform”): http://autumnai.com/

LightNet

LightNet: A Versatile, Standalone and Matlab-based Environment for Deep Learning

• homepage: http://www.umiacs.umd.edu/~yzyang/LightNet/
• github: https://github.com/yechengxi/lightnet

MatConvNet

MatConvNet: CNNs for MATLAB

• homepage: http://www.vlfeat.org/matconvnet/
• github: https://github.com/vlfeat/matconvnet

Marvin

Marvin: A minimalist GPU-only N-dimensional ConvNet framework

• homepage: http://marvin.is/
• github: https://github.com/PrincetonVision/marvin

MatConvNet: CNNs for MATLAB

• homepage: http://www.vlfeat.org/matconvnet/
• pretianed models: http://www.vlfeat.org/matconvnet/pretrained/

Mocha.jl

Mocha.jl: Deep Learning for Julia

• homepage: http://devblogs.nvidia.com/parallelforall/mocha-jl-deep-learning-julia/
• github: https://github.com/pluskid/Mocha.jl

MXNet

MXNet

• github: https://github.com/dmlc/mxnet

MXNet: A Flexible and Efficient Machine Learning Library for Heterogeneous Distributed Systems

• paper: https://raw.githubusercontent.com/dmlc/web-data/master/mxnet/paper/mxnet-learningsys.pdf

MXNet Model Gallery: Pre-trained Models of DMLC Project

• github: https://github.com/dmlc/mxnet-model-gallery

a short introduction to mxnet design and implementation (chinese)

• github: https://github.com/dmlc/mxnet/blob/master/doc/overview_chn.md
• github-issues: https://github.com/dmlc/mxnet/issues/797

Deep learning for hackers with MXnet (1) GPU installation and MNIST

https://no2147483647.wordpress.com/2015/12/07/deep-learning-for-hackers-with-mxnet-1/

mxnet_Efficient, Flexible Deep Learning Framework

• slides: http://vdisk.weibo.com/s/z5dg0jVVHv2pn/1450157571

Use Caffe operator in MXNet

• blog: http://dmlc.ml/mxnet/2016/07/29/use-caffe-operator-in-mxnet.html**

Deep Learning in a Single File for Smart Devices

https://mxnet.readthedocs.org/en/latest/tutorial/smart_device.html

MXNet Pascal Titan X benchmark

• blog: http://dmlc.ml/mxnet/2016/08/03/mxnet-titanx-benchmark.html

用MXnet实战深度学习之一:安装GPU版mxnet并跑一个MNIST手写数字识别

http://phunter.farbox.com/post/mxnet-tutorial1

用MXnet实战深度学习之二:Neural art

http://phunter.farbox.com/post/mxnet-tutorial2

Programming Models and Systems Design for Deep Learning

• video: http://research.microsoft.com/apps/video/default.aspx?id=262396
• video: http://pan.baidu.com/s/1mgSnj64

Awesome MXNet

• intro: This page contains a curated list of awesome MXnet examples, tutorials and blogs.
• github: https://github.com/dmlc/mxnet/blob/master/example/README.md

Getting Started with MXNet

https://indico.io/blog/getting-started-with-mxnet/

gtc_tutorial: MXNet Tutorial for NVidia GTC 2016

• report: http://on-demand.gputechconf.com/gtc/2016/video/S6853.html
• tutorial: http://on-demand.gputechconf.com/gtc/2016/video/L6143.html
• video: http://pan.baidu.com/s/1eS58Gue
• github: https://github.com/dmlc/mxnet-gtc-tutorial

MXNET Dependency Engine

• blog: http://yuyang0.github.io/articles/mxnet-engine.html

MXNET是这样压榨深度学习的内存消耗的

• doc: https://github.com/dmlc/mxnet/blob/master/docs/zh/note_memory.md

WhatsThis-iOS: MXNet WhatThis Example for iOS

• github: https://github.com/pppoe/WhatsThis-iOS

MXNET-MPI: Embedding MPI parallelism in Parameter Server Task Model for scaling Deep Learning

• intro: IBM T J Watson Research Center
• arxiv: https://arxiv.org/abs/1801.03855

ncnn

• intro: ncnn is a high-performance neural network inference framework optimized for the mobile platform
• github: https://github.com/Tencent/ncnn

neocortex.js

Run trained deep neural networks in the browser or node.js

• homepage: http://scienceai.github.io/neocortex/
• github: https://github.com/scienceai/neocortex

Neon

Neon: Nervana’s Python-based deep learning library

• website: http://neon.nervanasys.com/docs/latest/index.html
• github: https://github.com/NervanaSystems/neon
• website: https://www.nervanasys.com/learn/

Tools to convert Caffe models to neon’s serialization format

• github: https://github.com/NervanaSystems/caffe2neon

Nervana’s Deep Learning Course

• homepage: https://www.nervanasys.com/deep-learning-tutorials/
• github: https://github.com/NervanaSystems/neon_course

NNabla

NNabla - Neural Network Libraries by Sony

• intro: NNabla - Neural Network Libraries NNabla is a deep learning framework that is intended to be used for research, development and production. We aim it running everywhere like desktop PCs, HPC clusters, embedded devices and production servers.
• homepage: https://nnabla.org/
• github: https://github.com/sony/nnabla

OpenDeep

OpenDeep: a fully modular & extensible deep learning framework in Python

• intro: Modular & extensible deep learning framework built on Theano
• homepage: http://www.opendeep.org/
• github: https://github.com/vitruvianscience/opendeep

OpenNN

OpenNN - Open Neural Networks Library

• homepage: http://opennn.net/
• github: https://github.com/artelnics/opennn

Paddle

PaddlePaddle: PArallel Distributed Deep LEarning

• homepage: http://www.paddlepaddle.org/
• github: https://github.com/baidu/Paddle
• installation: http://www.paddlepaddle.org/doc/build/

基于Spark的异构分布式深度学习平台

http://geek.csdn.net/news/detail/58867

Petuum

Petuum: a distributed machine learning framework

• website: http://petuum.github.io/
• github: https://github.com/petuum/bosen

PlaidML

PlaidML: A framework for making deep learning work everywhere

• homepage: http://vertex.ai/
• github: https://github.com/plaidml/plaidml

Platoon

Platoon: Multi-GPU mini-framework for Theano

• github: https://github.com/mila-udem/platoon

Poseidon

Poseidon: Distributed Deep Learning Framework on Petuum

• github: https://github.com/petuum/poseidon
• wiki: https://github.com/petuum/poseidon/wiki

Purine

Purine: A bi-graph based deep learning framework

• github: https://github.com/purine/purine2
• arxiv: http://arxiv.org/abs/1412.6249

PyTorch

PyTorch

• github: https://github.com/pytorch/pytorch

Datasets, Transforms and Models specific to Computer Vision

https://github.com/pytorch/vision/

Convert torch to pytorch

https://github.com/clcarwin/convert_torch_to_pytorch

TensorFlow

TensorFlow

• website: http://tensorflow.org/
• github: https://github.com/tensorflow/tensorflow
• github: https://github.com/tensorflow/tensorflow/tree/master/tensorflow/core/distributed_runtime
• tutorial: http://tensorflow.org/tutorials
• tutorial: https://github.com/nlintz/TensorFlow-Tutorials
• stackoverflow: https://stackoverflow.com/questions/tagged/tensorflow
• benchmark: https://github.com/soumith/convnet-benchmarks/issues/66

Benchmarks

• intro: A selection of image classification models were tested across multiple platforms to create a point of reference for the TensorFlow community
• homepage: https://www.tensorflow.org/performance/benchmarks

TensorDebugger (TDB)

TensorDebugger(TDB): Interactive, node-by-node debugging and visualization for TensorFlow

• github: https://github.com/ericjang/tdb

ofxMSATensorFlow: OpenFrameworks addon for Google’s data-flow graph based numerical computation / machine intelligence library TensorFlow.

• github: https://github.com/memo/ofxMSATensorFlow

TFLearn: Deep learning library featuring a higher-level API for TensorFlow

• homepage: http://tflearn.org/
• github: https://github.com/tflearn/tflearn
• examples: https://github.com/tflearn/tflearn/blob/0.1.0/examples/README.md

TensorFlow on Spark

• github: https://github.com/adatao/tensorspark

TensorBoard

TensorFlow.jl: A Julia wrapper for the TensorFlow Python library

• github: https://github.com/benmoran/TensorFlow.jl

TensorLayer: Deep learning and Reinforcement learning library for TensorFlow

• github: https://github.com/zsdonghao/tensorlayer
• docs: http://tensorlayer.readthedocs.io/en/latest/

OpenCL support for TensorFlow

• github: https://github.com/benoitsteiner/tensorflow-opencl

Pretty Tensor: Fluent Networks in TensorFlow

• github: https://github.com/google/prettytensor
• docs: https://github.com/google/prettytensor/blob/master/docs/pretty_tensor_top_level.md
• tutorials: https://github.com/google/prettytensor/tree/master/prettytensor/tutorial

Rust language bindings for TensorFlow

• github: https://github.com/tensorflow/rust

TensorFlow Ecosystem: Integration of TensorFlow with other open-source frameworks

• github: https://github.com/tensorflow/ecosystem

Caffe to TensorFlow

• intro: Convert Caffe models to TensorFlow.
• github: https://github.com/ethereon/caffe-tensorflow

TensorFlow Mobile

https://www.tensorflow.org/mobile/

Papers

TensorFlow: Large-Scale Machine Learning on Heterogeneous Distributed Systems

• arxiv: http://arxiv.org/abs/1603.04467
• whitepaper: http://download.tensorflow.org/paper/whitepaper2015.pdf

TensorFlow: A system for large-scale machine learning

• arxiv: http://arxiv.org/abs/1605.08695

TensorFlow Distributions

https://arxiv.org/abs/1711.10604

Tutorials

TensorFlow 官方文档中文版

• tutorial-zh: https://github.com/jikexueyuanwiki/tensorflow-zh
• homepage: http://wiki.jikexueyuan.com/project/tensorflow-zh/

Theano

Theano

• website: http://deeplearning.net/software/theano/index.html
• github: https://github.com/Theano/Theano

Theano-Tutorials: Bare bones introduction to machine learning from linear regression to convolutional neural networks using Theano

• github: https://github.com/Newmu/Theano-Tutorials

Theano: A Python framework for fast computation of mathematical expressions

• arxiv: http://arxiv.org/abs/1605.02688

Configuring Theano For High Performance Deep Learning

http://www.johnwittenauer.net/configuring-theano-for-high-performance-deep-learning/

Theano: a short practical guide

• slides: http://folinoid.com/show/theano/

Ian Goodfellow’s Tutorials on Theano

• slides: http://pan.baidu.com/s/1slbzhF3#path=%252F%25E6%2588%2591%25E7%259A%2584%25E5%2588%2586%25E4%25BA%25AB%252F201604%252FIan%2520Goodfellow’s%2520Tutorials%2520on%2520Theano
• github(“theano_exercises”): https://github.com/goodfeli/theano_exercises

Plato: A library built on top of Theano

• github: https://github.com/petered/plato
• tutorial: https://rawgit.com/petered/plato/master/plato_tutorial.html

Theano Windows Install Guide

• github: https://github.com/mrakgr/Tutorials/blob/master/theano_install.md

Theano-MPI: a Theano-based Distributed Training Framework

• arxiv: https://arxiv.org/abs/1605.08325
• github: https://github.com/uoguelph-mlrg/Theano-MPI

tiny-dnn (tiny-cnn)

tiny-dnn: A header only, dependency-free deep learning framework in C++11

• inrtro: tiny-dnn is a C++11 implementation of deep learning. It is suitable for deep learning on limited computational resource, embedded systems and IoT devices.
• github: https://github.com/tiny-dnn/tiny-dnn
• github: https://github.com/nyanp/tiny-cnn

Deep learning with C++ - an introduction to tiny-dnn

• slides: http://www.slideshare.net/ssuser756ec5/deep-learning-with-c-an-introduction-to-tinydnn

Torch

Torch

• website: http://torch.ch/
• github: https://github.com/torch/torch7
• cheatsheet: https://github.com/torch/torch7/wiki/Cheatsheet
• tutorials(“Getting started with Torch”): [http://torch.ch/docs/getting-started.html#](http://torch.ch/docs/getting-started.html#)

loadcaffe: Load Caffe networks in Torch7

• github: https://github.com/szagoruyko/loadcaffe

Applied Deep Learning for Computer Vision with Torch

• homepage: https://github.com/soumith/cvpr2015

pytorch: Python wrappers for torch and lua

• github: https://github.com/hughperkins/pytorch

Torch Toolbox: A collection of snippets and libraries for Torch

• github: https://github.com/e-lab/torch-toolbox

cltorch: a Hardware-Agnostic Backend for the Torch Deep Neural Network Library, Based on OpenCL

• arxiv: http://arxiv.org/abs/1606.04884
• github: https://github.com/hughperkins/cltorch

Torchnet: An Open-Source Platform for (Deep) Learning Research

• paper: https://lvdmaaten.github.io/publications/papers/Torchnet_2016.pdf
• github: https://github.com/torchnet/torchnet

THFFmpeg: Torch bindings for FFmpeg (reading videos only)

• github: https://github.com/MichaelMathieu/THFFmpeg

caffegraph: Load Caffe networks in Torch7 using nngraph

• github: https://github.com/nhynes/caffegraph

Optimized-Torch: Intel Torch is dedicated to improving Torch performance when running on CPU

• intro: Intel Torch gets 4.66x speedup using the convnet-benchmarks which includes AlexNet,VGG-E,GoogLenet,ResidualNet
• github: https://github.com/xhzhao/optimized-torch
• benchmark: https://github.com/xhzhao/Optimized-Torch-benchmark

Torch Video Tutorials

• github: https://github.com/Atcold/torch-Video-Tutorials

Torch in Action

• github: https://github.com/nicholas-leonard/torch-in-action

VELES

VELES: Distributed platform for rapid Deep learning application development

• website: https://velesnet.ml/
• github: https://github.com/Samsung/veles
• workflow: https://velesnet.ml/forge/forge.html

WebDNN

WebDNN: Fastest DNN Execution Framework on Web Browser

• homepage: https://mil-tokyo.github.io/webdnn/
• github: https://github.com/mil-tokyo/webdnn

Yann

Yann: Yet Another Neural Network Toolbox

• intro: It is a toolbox for building and learning convolutional neural networks, built on top of theano
• github: https://github.com/ragavvenkatesan/yann
• docs: http://yann.readthedocs.io/en/master/

Benchmarks

Easy benchmarking of all publicly accessible implementations of convnets

https://github.com/soumith/convnet-benchmarks

Stanford DAWN Deep Learning Benchmark (DAWNBench) - An End-to-End Deep Learning Benchmark and Competition

http://dawn.cs.stanford.edu/benchmark/index.html

Tutorials

Deep Learning Implementations and Frameworks (DLIF)

• tutorial: https://sites.google.com/site/dliftutorial/
• github: https://github.com/delta2323/DLIF-tutorial

Papers

Comparative Study of Deep Learning Software Frameworks

• intro: Caffe / Neon / TensorFlow / Theano / Torch
• arxiv: http://arxiv.org/abs/1511.06435
• github: https://github.com/DL-Benchmarks/DL-Benchmarks

Benchmarking State-of-the-Art Deep Learning Software Tools

• intro: Caffe, CNTK, MXNet, TensorFlow, and Torch
• project page: http://dlbench.comp.hkbu.edu.hk/
• arxiv: http://arxiv.org/abs/1608.07249

Projects

TensorFuse: Common interface for Theano, CGT, and TensorFlow

• github: https://github.com/dementrock/tensorfuse

DeepRosetta: An universal deep learning models conversor

• github: https://github.com/edgarriba/DeepRosetta

Deep Learning Model Convertors

https://github.com/ysh329/deep-learning-model-convertor

References

Frameworks and Libraries for Deep Learning

http://creative-punch.net/2015/07/frameworks-and-libraries-for-deep-learning/

TensorFlow vs. Theano vs. Torch

https://github.com/zer0n/deepframeworks/blob/master/README.md

Evaluation of Deep Learning Toolkits

https://github.com/zer0n/deepframeworks/blob/master/README.md

Deep Machine Learning libraries and frameworks

https://medium.com/@abduljaleel/deep-machine-learning-libraries-and-frameworks-5fdf2bb6bfbe#.q1mhj7c36

Torch vs Theano

• blog: http://fastml.com/torch-vs-theano/

Deep Learning Software: NVIDIA Deep Learning SDK

https://developer.nvidia.com/deep-learning-software

A comparison of deep learning frameworks

• intro: Theano/CGT/Torch/MXNet
• gist: https://gist.github.com/bartvm/69adf7aad100d58831b0
• webo: http://weibo.com/p/1001603946281180481229

TensorFlow Meets Microsoft’s CNTK

• blog: http://esciencegroup.com/2016/02/08/tensorflow-meets-microsofts-cntk/

Is there a case for still using Torch, Theano, Brainstorm, MXNET and not switching to TensorFlow?

• reddit: [https://www.reddit.com/r/MachineLearning/comments/47qh90/is_there_a_case_for_still_using_torch_theano/][https://www.reddit.com/r/MachineLearning/comments/47qh90/is_there_a_case_for_still_using_torch_theano/]

DL4J vs. Torch vs. Theano vs. Caffe vs. TensorFlow

http://deeplearning4j.org/compare-dl4j-torch7-pylearn.html

Popular Deep Learning Libraries

• blog: http://machinelearningmastery.com/popular-deep-learning-libraries/

The simple example of Theano and Lasagne super power

https://grzegorzgwardys.wordpress.com/2016/05/15/the-simple-example-of-theano-and-lasagne-super-power/

Comparison of deep learning software

• wiki: https://en.wikipedia.org/wiki/Comparison_of_deep_learning_software

A Look at Popular Machine Learning Frameworks

• blog: http://redmonk.com/fryan/2016/06/06/a-look-at-popular-machine-learning-frameworks/

5 Deep Learning Projects You Can No Longer Overlook

• keywords: Leaf / tiny-cnn / Layered / Brain / neon
• blog: http://www.kdnuggets.com/2016/07/five-deep-learning-projects-cant-overlook.html

Comparison of Deep Learning Libraries After Years of Use

• intro: Torch / MxNet / Theano / Caffe
• blog:http://www.erogol.com/comparison-deep-learning-libraries-years-use/

Deep Learning Part 1: Comparison of Symbolic Deep Learning Frameworks

• intro: Theano / TensorFlow / MXNET
• blog: http://blog.revolutionanalytics.com/2016/08/deep-learning-part-1.html

Deep Learning Frameworks Compared

• youtube: https://www.youtube.com/watch?v=MDP9FfsNx60
• github: https://github.com/llSourcell/tensorflow_vs_theano

DL4J vs. Torch vs. Theano vs. Caffe vs. TensorFlow

https://deeplearning4j.org/compare-dl4j-torch7-pylearn.html

Deep Learning frameworks: a review before finishing 2016

https://medium.com/@ricardo.guerrero/deep-learning-frameworks-a-review-before-finishing-2016-5b3ab4010b06#.a6fdrqssl

The Anatomy of Deep Learning Frameworks

https://medium.com/@gokul_uf/the-anatomy-of-deep-learning-frameworks-46e2a7af5e47

Python Deep Learning Frameworks Reviewed

https://indico.io/blog/python-deep-learning-frameworks-reviewed/

Apple’s deep learning frameworks: BNNS vs. Metal CNN

http://machinethink.net/blog/apple-deep-learning-bnns-versus-metal-cnn/

Deep Learning

EECS 598: Unsupervised Feature Learning

• instructor: Honglak Lee
• homepage: http://web.eecs.umich.edu/~honglak/teaching/eecs598/schedule.html

NVIDIA’s Deep Learning Courses

https://developer.nvidia.com/deep-learning-courses

ECE 6504 Deep Learning for Perception

• instructor: Dhruv Batra (Virginia Tech)
• homepage: https://computing.ece.vt.edu/~f15ece6504/

University of Oxford: Machine Learning: 2014-2015

• homepage: https://www.cs.ox.ac.uk/people/nando.defreitas/machinelearning/
• lectures: http://pan.baidu.com/s/1bndbxJh#path=%252FDeep%2520Learning%2520Lectures
• github: https://github.com/oxford-cs-ml-2015/

University of Birmingham 2014: Introduction to Neural Computation (Level 4/M); Neural Computation (Level 3/H)(by John A. Bullinaria)

http://www.cs.bham.ac.uk/~jxb/inc.html

CMU: Deep Learning

• instructor: Bhiksha Raj
• homepage: http://deeplearning.cs.cmu.edu/

stat212b: Topics Course on Deep Learning for Spring 2016

• homepage: http://joanbruna.github.io/stat212b/
• github: https://github.com/joanbruna/stat212b
• pan: http://pan.baidu.com/s/1sk7TKtf#path=%252Fstat212b%2520-%2520Topics%2520Course%2520on%2520Deep%2520Learning%2520for%2520Spring%25202016

Good materials on deep learning

http://eclass.cc/courselists/117_deep_learning

Deep Learning: Course by Yann LeCun at Collège de France 2016(Slides in English)

• homepage: https://www.facebook.com/yann.lecun/posts/10153505343037143
• downloads: https://drive.google.com/open?id=0BxKBnD5y2M8NclFWSXNxa0JlZTg

CSC321 Winter 2015: Introduction to Neural Networks

• homepage: http://www.cs.toronto.edu/~rgrosse/csc321/calendar.html

ELEG 5040: Advanced Topics in Signal Processing (Introduction to Deep Learning)

• instructors: Xiaogang Wang. The Chinese University of Hong Kong - Spring 2015
• intro: Homework, Homework Solutions, Lecture Notes, General Resources, Tutorial Notes, CUDA/GPU programming tutorial
• homepage: https://piazza.com/cuhk.edu.hk/spring2015/eleg5040/resources

Self-Study Courses for Deep Learning (NVIDIA Deep Learning Institute)

• homepage: https://developer.nvidia.com/deep-learning-courses

Introduction to Deep Learning

• homepage: https://beta.bigdatauniversity.com/courses/introduction-deep-learning/

Deep Learning Courses

• blog: http://machinelearningmastery.com/deep-learning-courses/

Creative Applications of Deep Learning w/ Tensorflow

• homepage: https://www.kadenze.com/courses/creative-applications-of-deep-learning-with-tensorflow-i/info
• github(ourse materials/Homework materials): https://github.com/pkmital/CADL

Deep Learning School: September 24-25, 2016 Stanford, CA

• homepage: http://www.bayareadlschool.org/
• day 1: https://www.youtube.com/watch?v=9dXiAecyJrY
• day 2: https://www.youtube.com/watch?v=eyovmAtoUx0
• github: https://github.com/lamblin/bayareadlschool
• reddit: https://amp.reddit.com/r/MachineLearning/comments/54shmi/great_new_introductory_talks_on_various_subfields/
• mirror: https://pan.baidu.com/s/1gfBe2fL

CSC 2541 Fall 2016: Differentiable Inference and Generative Models

• homepage: http://www.cs.toronto.edu/~duvenaud/courses/csc2541/index.html

CS 294-131: Special Topics in Deep Learning (Fall, 2016)

https://berkeley-deep-learning.github.io/cs294-dl-f16/

Fork of Lempitsky DL for HSE master students.

• github: https://github.com/yandexdataschool/HSE_deeplearning

ELEG 5040: Advanced Topics in Signal Processing (Introduction to Deep Learning)

• resources: https://piazza.com/cuhk.edu.hk/spring2015/eleg5040/resources

CS 20SI: Tensorflow for Deep Learning Research

• homepage: http://web.stanford.edu/class/cs20si/
• github: https://github.com/chiphuyen/stanford-tensorflow-tutorials

Deep Learning with TensorFlow

https://bigdatauniversity.com/courses/deep-learning-tensorflow/

Deep Learning course

• github: https://github.com/ddtm/dl-course

CSE 599G1: Deep Learning System

• homepage: http://dlsys.cs.washington.edu/
• assignments: http://dlsys.cs.washington.edu/assignments

CSC 321 Winter 2017: Intro to Neural Networks and Machine Learning

http://www.cs.toronto.edu/~rgrosse/courses/csc321_2017/

Theories of Deep Learning (STATS 385)

• homepage: https://stats385.github.io/
• video: https://www.researchgate.net/project/Theories-of-Deep-Learning
• mirror: https://www.bilibili.com/video/av16136625/

CS230: Deep Learning Spring 2018

https://web.stanford.edu/class/cs230/

With Video Lectures

Deep Learning: Taking machine learning to the next level (Udacity)

• instructor: Vincent Vanhoucke (Google), Arpan Chakraborty
• homepage: https://www.udacity.com/course/deep-learning–ud730
• homepage: https://cn.udacity.com/course/deep-learning–ud730/
• homepage: https://classroom.udacity.com/courses/ud730/lessons/6370362152/concepts/63798118150923
• assignments: https://github.com/tdhopper/udacity-deep-learning
• ipn: https://github.com/tensorflow/tensorflow/blob/master/tensorflow/examples/udacity/1_notmnist.ipynb
• ipn: http://nbviewer.jupyter.org/github/tensorflow/tensorflow/blob/master/tensorflow/examples/udacity/1_notmnist.ipynb
• assignments: https://github.com/Arn-O/udacity-deep-learning

Neural networks class - Université de Sherbrooke

• instructor: Hugo Larochelle
• youtube: https://www.youtube.com/playlist?list=PL6Xpj9I5qXYEcOhn7TqghAJ6NAPrNmUBH
• video: http://pan.baidu.com/s/1bnwEe8R
• course content: http://info.usherbrooke.ca/hlarochelle/neural_networks/content.html
• google group: https://groups.google.com/forum/#!forum/neural-networks-online-course

Deep Learning: Theoretical Motivations

• author: Yoshua Bengio
• published: Sept. 13, 2015. (Deep Learning Summer School, Montreal 2015)
• video: http://videolectures.net/deeplearning2015_bengio_theoretical_motivations/
• blog: http://rinuboney.github.io/2015/10/18/theoretical-motivations-deep-learning.html

University of Waterloo: STAT 946 - Deep Learning

• homepage: https://uwaterloo.ca/data-science/deep-learning
• video+slides: http://pan.baidu.com/s/1sjTRgjN

Deep Learning (2016) - BME 595A, Eugenio Culurciello, Purdue University

• course shedule: http://t.cn/RVYQa69?u=1402400261&m=4034720314226808&cu=2261580215&ru=1402400261&rm=4034708389597157
• mirror: https://pan.baidu.com/s/1hsBJOpQ
• video: https://www.youtube.com/playlist?list=PLNgy4gid0G9cbw5OjwG2jxvFqYDqkGnpJ
• mirror: https://pan.baidu.com/s/1bpKb5Cj

UVA DEEP LEARNING COURSE

• intro: MSc in Artificial Intelligence for the University of Amsterdam.
• homepage: http://uvadlc.github.io/
• assignments: https://github.com/uvadlc/uvadlc_practicals_2016

Practical Deep Learning For Coders, Part 1

• intro: 10 hours a week for 7 weeks
• homepage: http://course.fast.ai/
• youtube: https://www.youtube.com/playlist?list=PLfYUBJiXbdtS2UQRzyrxmyVHoGW0gmLSM
• mirror: https://pan.baidu.com/s/1eRLK742#list/path=%2F
• github: https://github.com/fastai/courses
• blog: http://www.kdnuggets.com/2016/12/deep-learning-coders-mooc-jeremy-howard.html

T81-558:Applications of Deep Neural Networks

• intro: Washington University
• course page: https://sites.wustl.edu/jeffheaton/t81-558/
• youtube: https://www.youtube.com/playlist?list=PLjy4p-07OYzulelvJ5KVaT2pDlxivl_BN
• github: https://github.com/jeffheaton/t81_558_deep_learning

CS294-129 Designing, Visualizing and Understanding Deep Neural Networks

• homepage: https://bcourses.berkeley.edu/courses/1453965/pages/cs294-129-designing-visualizing-and-understanding-deep-neural-networks
• lecture video: https://www.youtube.com/playlist?list=PLkFD6_40KJIxopmdJF_CLNqG3QuDFHQUm

MIT 6.S191: Introduction to Deep Learning

• homepage: http://introtodeeplearning.com/index.html
• schedule(Slides+Videos): http://introtodeeplearning.com/schedule.html
• github: https://github.com/yala/introdeeplearning
• youtube: https://www.youtube.com/playlist?list=PLkkuNyzb8LmxFutYuPA7B4oiMn6cjD6Rs
• mirror: https://pan.baidu.com/s/1qXXDCoG#list/path=%2F

Edx: Deep Learning Explained

• intro: Microsoft
• course page: https://www.edx.org/course/deep-learning-explained-microsoft-dat236x

Computer Vision

Stanford CS231n: Convolutional Neural Networks for Visual Recognition (Spring 2017)

• youtube: https://www.youtube.com/playlist?list=PL3FW7Lu3i5JvHM8ljYj-zLfQRF3EO8sYv
• mirror: http://www.bilibili.com/video/av13260183/

Stanford CS231n: Convolutional Neural Networks for Visual Recognition (Winter 2016)

• homepage: http://cs231n.stanford.edu/
• homepage: http://vision.stanford.edu/teaching/cs231n/index.html
• syllabus: http://vision.stanford.edu/teaching/cs231n/syllabus.html
• course notes: http://cs231n.github.io/
• youtube: https://www.youtube.com/watch?v=NfnWJUyUJYU&feature=youtu.be
• mirror: http://pan.baidu.com/s/1pKsTivp
• mirror: http://pan.baidu.com/s/1c2wR8dy
• assignment 1: http://cs231n.github.io/assignments2016/assignment1/
• assignment 2: http://cs231n.github.io/assignments2016/assignment2/
• assignment 3: http://cs231n.github.io/assignments2016/assignment3/

ITP-NYU - Spring 2016

• Video lectures and course notes: http://ml4a.github.io/classes/itp-S16/

Deep Learning for Computer Vision Barcelona: Summer seminar UPC TelecomBCN (July 4-8, 2016)

• intro: This course will cover the basic principles and applications of deep learning to computer vision problems, such as image classification, object detection or text captioning.
• homepage(slides+videos): http://imatge-upc.github.io/telecombcn-2016-dlcv/
• homepage: https://imatge.upc.edu/web/teaching/deep-learning-computer-vision
• youtube: https://www.youtube.com/user/imatgeupc/videos?shelf_id=0&sort=dd&view=0

DLCV - Deep Learning for Computer Vision

• homepage: https://imatge.upc.edu/web/teaching/deep-learning-computer-vision

Advanced Computer Vision Cap6412

• homepage: http://crcv.ucf.edu/courses/CAP6412/Spring2018/
• video: https://www.youtube.com/playlist?list=PLd3hlSJsX_ImoNaeX5vFrxogGXTSmS993

Natural Language Processing

CS224n: Natural Language Processing with Deep Learning

• intro: This course is a merger of Stanford’s previous cs224n course and cs224d
• homepage: http://web.stanford.edu/class/cs224n/

Course notes for CS224N Winter17

https://github.com/stanfordnlp/cs224n-winter17-notes

Stanford CS224d: Deep Learning for Natural Language Processing

• homepage: http://cs224d.stanford.edu/
• syllabus: http://cs224d.stanford.edu/syllabus.html
• lecture notes: https://cs224d.stanford.edu/lecture_notes/

Code for Stanford CS224D: deep learning for natural language understanding

• github: https://github.com/bogatyy/cs224d

CMU CS 11-747, Fall 2017: Neural Networks for NLP

• intro: by Graham Neubig
• course page: http://phontron.com/class/nn4nlp2017/
• github: https://github.com/neubig/nn4nlp2017-code
• video: https://www.bilibili.com/video/av14153689/

Deep Learning for NLP - Lecture October 2015

• github: https://github.com/UKPLab/deeplearning4nlp-tutorial/tree/master/2015-10_Lecture

Harvard University: CS287: Natural Language Processing

http://cs287.fas.harvard.edu/

Deep Learning for Natural Language Processing: 2016-2017

• intro: Oxford Deep NLP 2017 course
• homepage: http://www.cs.ox.ac.uk/teaching/courses/2016-2017/dl/
• github: https://github.com/oxford-cs-deepnlp-2017/lectures
• youtube: https://www.youtube.com/playlist?list=PL613dYIGMXoZBtZhbyiBqb0QtgK6oJbpm
• mirror: https://pan.baidu.com/s/1dFvGHUh#list/path=%2F
• mirror: https://pan.baidu.com/s/1c2tcC96

GPU Programming

Course on CUDA Programming on NVIDIA GPUs, July 27–31, 2015

• homepage: http://people.maths.ox.ac.uk/gilesm/cuda/

An Introduction to GPU Programming using Theano

• youtube: https://www.youtube.com/watch?v=eVd2TqEkVp0
• video: http://pan.baidu.com/s/1c1i6LtI#path=%252F

GPU Programming

• homepage: http://courses.cms.caltech.edu/cs179/

Parallel Programming

Intro to Parallel Programming Using CUDA to Harness the Power of GPUs (Udacity)

https://www.udacity.com/course/intro-to-parallel-programming–cs344

Fundamentals of Accelerated Computing with CUDA C/C++

• intro: Learn to use CUDA C/C++ tools and techniques to accelerate CPU-only applications to run on massively parallel GPUs.
• homepage: https://courses.nvidia.com/courses/course-v1:DLI+C-AC-01+V1/about

Workshops

Deep Learning: Theory, Algorithms, and Applications

• homepage: http://doc.ml.tu-berlin.de/dlworkshop2017/
• video: https://www.youtube.com/playlist?list=PLJOzdkh8T5kqCNV_v1w2tapvtJDZYiohW
• mirror: https://www.bilibili.com/video/av15565354/

Resources

Open Source Deep Learning Curriculum

http://www.deeplearningweekly.com/pages/open_source_deep_learning_curriculum

Applications

Papers

Papers

Im2Text: Describing Images Using 1 Million Captioned Photographs

• paper: http://tamaraberg.com/papers/generation_nips2011.pdf
• project: http://vision.cs.stonybrook.edu/~vicente/sbucaptions/

Long-term Recurrent Convolutional Networks for Visual Recognition and Description

• intro: Oral presentation at CVPR 2015. LRCN
• project page: http://jeffdonahue.com/lrcn/
• arxiv: http://arxiv.org/abs/1411.4389
• github: https://github.com/BVLC/caffe/pull/2033

Show and Tell

Show and Tell: A Neural Image Caption Generator

• intro: Google
• arxiv: http://arxiv.org/abs/1411.4555
• github: https://github.com/karpathy/neuraltalk
• gitxiv: http://gitxiv.com/posts/7nofxjoYBXga5XjtL/show-and-tell-a-neural-image-caption-nic-generator
• github: https://github.com/apple2373/chainer_caption_generation
• github(TensorFlow): https://github.com/tensorflow/models/tree/master/im2txt
• github(TensorFlow): https://github.com/zsdonghao/Image-Captioning

Image caption generation by CNN and LSTM

• blog: http://t-satoshi.blogspot.com/2015/12/image-caption-generation-by-cnn-and-lstm.html
• github: https://github.com/jazzsaxmafia/show_and_tell.tensorflow

Show and Tell: Lessons learned from the 2015 MSCOCO Image Captioning Challenge

• arxiv: http://arxiv.org/abs/1609.06647
• github: https://github.com/tensorflow/models/tree/master/im2txt

Learning a Recurrent Visual Representation for Image Caption Generation

• arxiv: http://arxiv.org/abs/1411.5654

Mind’s Eye: A Recurrent Visual Representation for Image Caption Generation

• intro: CVPR 2015
• paper: http://www.cs.cmu.edu/~xinleic/papers/cvpr15_rnn.pdf

Deep Visual-Semantic Alignments for Generating Image Descriptions

• intro: “propose a multimodal deep network that aligns various interesting regions of the image, represented using a CNN feature, with associated words. The learned correspondences are then used to train a bi-directional RNN. This model is able, not only to generate descriptions for images, but also to localize different segments of the sentence to their corresponding image regions.”
• project page: http://cs.stanford.edu/people/karpathy/deepimagesent/
• arxiv: http://arxiv.org/abs/1412.2306
• slides: http://www.cs.toronto.edu/~vendrov/DeepVisualSemanticAlignments_Class_Presentation.pdf
• github: https://github.com/karpathy/neuraltalk
• demo: http://cs.stanford.edu/people/karpathy/deepimagesent/rankingdemo/

Deep Captioning with Multimodal Recurrent Neural Networks

• intro: m-RNN. ICLR 2015
• intro: “combines the functionalities of the CNN and RNN by introducing a new multimodal layer, after the embedding and recurrent layers of the RNN.”
• homepage: http://www.stat.ucla.edu/~junhua.mao/m-RNN.html
• arxiv: http://arxiv.org/abs/1412.6632
• github: https://github.com/mjhucla/mRNN-CR
• github: https://github.com/mjhucla/TF-mRNN

Show, Attend and Tell

Show, Attend and Tell: Neural Image Caption Generation with Visual Attention (ICML 2015)

• project page: http://kelvinxu.github.io/projects/capgen.html
• arxiv: http://arxiv.org/abs/1502.03044
• github: https://github.com/kelvinxu/arctic-captions
• github: https://github.com/jazzsaxmafia/show_attend_and_tell.tensorflow
• github(TensorFlow): https://github.com/yunjey/show-attend-and-tell-tensorflow
• demo: http://www.cs.toronto.edu/~rkiros/abstract_captions.html

Automatically describing historic photographs

• website: https://staff.fnwi.uva.nl/d.elliott/loc/

Learning like a Child: Fast Novel Visual Concept Learning from Sentence Descriptions of Images

• arxiv: http://arxiv.org/abs/1504.06692
• homepage: http://www.stat.ucla.edu/~junhua.mao/projects/child_learning.html
• github: https://github.com/mjhucla/NVC-Dataset

What value do explicit high level concepts have in vision to language problems?

• arxiv: http://arxiv.org/abs/1506.01144

Aligning where to see and what to tell: image caption with region-based attention and scene factorization

• arxiv: http://arxiv.org/abs/1506.06272

Learning FRAME Models Using CNN Filters for Knowledge Visualization (CVPR 2015)

• project page: http://www.stat.ucla.edu/~yang.lu/project/deepFrame/main.html
• arxiv: http://arxiv.org/abs/1509.08379
• code+data: http://www.stat.ucla.edu/~yang.lu/project/deepFrame/doc/deepFRAME_1.1.zip

Generating Images from Captions with Attention

• arxiv: http://arxiv.org/abs/1511.02793
• github: https://github.com/emansim/text2image
• demo: http://www.cs.toronto.edu/~emansim/cap2im.html

Order-Embeddings of Images and Language

• arxiv: http://arxiv.org/abs/1511.06361
• github: https://github.com/ivendrov/order-embedding

DenseCap: Fully Convolutional Localization Networks for Dense Captioning

• project page: http://cs.stanford.edu/people/karpathy/densecap/
• arxiv: http://arxiv.org/abs/1511.07571
• github(Torch): https://github.com/jcjohnson/densecap

Expressing an Image Stream with a Sequence of Natural Sentences

• intro: NIPS 2015. CRCN
• nips-page: http://papers.nips.cc/paper/5776-expressing-an-image-stream-with-a-sequence-of-natural-sentences
• paper: http://papers.nips.cc/paper/5776-expressing-an-image-stream-with-a-sequence-of-natural-sentences.pdf
• paper: http://www.cs.cmu.edu/~gunhee/publish/nips15_stream2text.pdf
• author-page: http://www.cs.cmu.edu/~gunhee/
• github: https://github.com/cesc-park/CRCN

Multimodal Pivots for Image Caption Translation

• intro: ACL 2016
• arxiv: http://arxiv.org/abs/1601.03916

Image Captioning with Deep Bidirectional LSTMs

• intro: ACMMM 2016
• arxiv: http://arxiv.org/abs/1604.00790
• github(Caffe): https://github.com/deepsemantic/image_captioning
• demo: https://youtu.be/a0bh9_2LE24

Encode, Review, and Decode: Reviewer Module for Caption Generation

Review Network for Caption Generation

• intro: NIPS 2016
• arxiv: https://arxiv.org/abs/1605.07912
• github: https://github.com/kimiyoung/review_net

Attention Correctness in Neural Image Captioning

• arxiv: http://arxiv.org/abs/1605.09553

Image Caption Generation with Text-Conditional Semantic Attention

• arxiv: https://arxiv.org/abs/1606.04621
• github: https://github.com/LuoweiZhou/e2e-gLSTM-sc

DeepDiary: Automatic Caption Generation for Lifelogging Image Streams

• intro: ECCV International Workshop on Egocentric Perception, Interaction, and Computing
• arxiv: http://arxiv.org/abs/1608.03819

phi-LSTM: A Phrase-based Hierarchical LSTM Model for Image Captioning

• intro: ACCV 2016
• arxiv: http://arxiv.org/abs/1608.05813

Captioning Images with Diverse Objects

• arxiv: http://arxiv.org/abs/1606.07770

Learning to generalize to new compositions in image understanding

• arxiv: http://arxiv.org/abs/1608.07639

Generating captions without looking beyond objects

• intro: ECCV2016 2nd Workshop on Storytelling with Images and Videos (VisStory)
• arxiv: https://arxiv.org/abs/1610.03708

SPICE: Semantic Propositional Image Caption Evaluation

• intro: ECCV 2016
• project page: http://www.panderson.me/spice/
• paper: http://www.panderson.me/images/SPICE.pdf
• github: https://github.com/peteanderson80/SPICE

Boosting Image Captioning with Attributes

• arxiv: https://arxiv.org/abs/1611.01646

Bootstrap, Review, Decode: Using Out-of-Domain Textual Data to Improve Image Captioning

• arxiv: https://arxiv.org/abs/1611.05321

A Hierarchical Approach for Generating Descriptive Image Paragraphs

• intro: Stanford University
• arxiv: https://arxiv.org/abs/1611.06607

Dense Captioning with Joint Inference and Visual Context

• intro: Snap Inc.
• arxiv: https://arxiv.org/abs/1611.06949

Optimization of image description metrics using policy gradient methods

• intro: University of Oxford & Google
• arxiv: https://arxiv.org/abs/1612.00370

Areas of Attention for Image Captioning

• arxiv: https://arxiv.org/abs/1612.01033

Knowing When to Look: Adaptive Attention via A Visual Sentinel for Image Captioning

• intro: CVPR 2017
• arxiv: https://arxiv.org/abs/1612.01887
• github: https://github.com/jiasenlu/AdaptiveAttention

Recurrent Image Captioner: Describing Images with Spatial-Invariant Transformation and Attention Filtering

• arxiv: https://arxiv.org/abs/1612.04949

Recurrent Highway Networks with Language CNN for Image Captioning

• arxiv: https://arxiv.org/abs/1612.07086

Top-down Visual Saliency Guided by Captions

• arxiv: https://arxiv.org/abs/1612.07360
• github: https://github.com/VisionLearningGroup/caption-guided-saliency

MAT: A Multimodal Attentive Translator for Image Captioning

https://arxiv.org/abs/1702.05658

Deep Reinforcement Learning-based Image Captioning with Embedding Reward

• intro: Snap Inc & Google Inc
• arxiv: https://arxiv.org/abs/1704.03899

Attend to You: Personalized Image Captioning with Context Sequence Memory Networks

• intro: CVPR 2017
• arxiv: https://arxiv.org/abs/1704.06485
• github: https://github.com/cesc-park/attend2u

Punny Captions: Witty Wordplay in Image Descriptions

https://arxiv.org/abs/1704.08224

Show, Adapt and Tell: Adversarial Training of Cross-domain Image Captioner

https://arxiv.org/abs/1705.00930

Actor-Critic Sequence Training for Image Captioning

• intro: Queen Mary University of London & Yang’s Accounting Consultancy Ltd
• keywords: actor-critic reinforcement learning
• arxiv: https://arxiv.org/abs/1706.09601

What is the Role of Recurrent Neural Networks (RNNs) in an Image Caption Generator?

• intro: Proceedings of the 10th International Conference on Natural Language Generation (INLG’17)
• arxiv: https://arxiv.org/abs/1708.02043

Stack-Captioning: Coarse-to-Fine Learning for Image Captioning

https://arxiv.org/abs/1709.03376

Self-Guiding Multimodal LSTM - when we do not have a perfect training dataset for image captioning

https://arxiv.org/abs/1709.05038

Contrastive Learning for Image Captioning

• intro: NIPS 2017
• arxiv: https://arxiv.org/abs/1710.02534

Phrase-based Image Captioning with Hierarchical LSTM Model

• intro: ACCV2016 extension, phrase-based image captioning
• arxiv: https://arxiv.org/abs/1711.05557

Convolutional Image Captioning

https://arxiv.org/abs/1711.09151

Show-and-Fool: Crafting Adversarial Examples for Neural Image Captioning

https://arxiv.org/abs/1712.02051

Improved Image Captioning with Adversarial Semantic Alignment

• intro: IBM Research
• arxiv: https://arxiv.org/abs/1805.00063

Object Counts! Bringing Explicit Detections Back into Image Captioning

• intro: NAACL 2018
• arxiv: https://arxiv.org/abs/1805.00314

Defoiling Foiled Image Captions

• intro: NAACL 2018
• arxiv: https://arxiv.org/abs/1805.06549

SemStyle: Learning to Generate Stylised Image Captions using Unaligned Text

• intro: CVPR 2018
• arxiv: https://arxiv.org/abs/1805.07030

Improving Image Captioning with Conditional Generative Adversarial Nets

https://arxiv.org/abs/1805.07112

CNN+CNN: Convolutional Decoders for Image Captioning

https://arxiv.org/abs/1805.09019

Diverse and Controllable Image Captioning with Part-of-Speech Guidance

https://arxiv.org/abs/1805.12589

Learning to Evaluate Image Captioning

• intro: CVPR 2018
• arxiv: https://arxiv.org/abs/1806.06422

Topic-Guided Attention for Image Captioning

• intro: ICIP 2018
• arxiv: https://arxiv.org/abs/1807.03514

Context-Aware Visual Policy Network for Sequence-Level Image Captioning

• intro: ACM MM 2018 oral
• arxiv: https://arxiv.org/abs/1808.05864
• github: https://github.com/daqingliu/CAVP

Exploring Visual Relationship for Image Captioning

• intro: ECCV 2018
• arxiv: https://arxiv.org/abs/1809.07041

Boosted Attention: Leveraging Human Attention for Image Captioning

• intro: ECCV 2018
• arxiv: https://arxiv.org/abs/1904.00767

Image Captioning as Neural Machine Translation Task in SOCKEYE

https://arxiv.org/abs/1810.04101

Unsupervised Image Captioning

https://arxiv.org/abs/1811.10787

Attend More Times for Image Captioning

https://arxiv.org/abs/1812.03283

Object Descriptions

Generation and Comprehension of Unambiguous Object Descriptions

• arxiv: https://arxiv.org/abs/1511.02283
• github: https://github.com/mjhucla/Google_Refexp_toolbox

Video Captioning / Description

Jointly Modeling Deep Video and Compositional Text to Bridge Vision and Language in a Unified Framework

• intro: AAAI 2015
• paper: http://www.cv-foundation.org/openaccess/content_cvpr_2016/papers/Pan_Jointly_Modeling_Embedding_CVPR_2016_paper.pdf
• paper: http://web.eecs.umich.edu/~jjcorso/pubs/xu_corso_AAAI2015_v2t.pdf

Translating Videos to Natural Language Using Deep Recurrent Neural Networks

• intro: NAACL-HLT 2015 camera ready
• project page: https://www.cs.utexas.edu/~vsub/naacl15_project.html
• arxiv: http://arxiv.org/abs/1412.4729
• slides: https://www.cs.utexas.edu/~vsub/pdf/Translating_Videos_slides.pdf
• code+data: https://www.cs.utexas.edu/~vsub/naacl15_project.html#code

Describing Videos by Exploiting Temporal Structure

• arxiv: http://arxiv.org/abs/1502.08029
• github: https://github.com/yaoli/arctic-capgen-vid

SA-tensorflow: Soft attention mechanism for video caption generation

• github: https://github.com/tsenghungchen/SA-tensorflow

Sequence to Sequence – Video to Text

• intro: ICCV 2015. S2VT
• project page: http://vsubhashini.github.io/s2vt.html
• arxiv: http://arxiv.org/abs/1505.00487
• slides: https://www.cs.utexas.edu/~vsub/pdf/S2VT_slides.pdf
• github(Caffe): https://github.com/vsubhashini/caffe/tree/recurrent/examples/s2vt
• github(TensorFlow): https://github.com/jazzsaxmafia/video_to_sequence

Jointly Modeling Embedding and Translation to Bridge Video and Language

• arxiv: http://arxiv.org/abs/1505.01861

Video Description using Bidirectional Recurrent Neural Networks

• arxiv: http://arxiv.org/abs/1604.03390

Bidirectional Long-Short Term Memory for Video Description

• arxiv: https://arxiv.org/abs/1606.04631

3 Ways to Subtitle and Caption Your Videos Automatically Using Artificial Intelligence

• blog: http://photography.tutsplus.com/tutorials/3-ways-to-subtitle-and-caption-your-videos-automatically-using-artificial-intelligence–cms-26834

Frame- and Segment-Level Features and Candidate Pool Evaluation for Video Caption Generation

• arxiv: http://arxiv.org/abs/1608.04959

Grounding and Generation of Natural Language Descriptions for Images and Videos

• intro: Anna Rohrbach. Allen Institute for Artificial Intelligence (AI2)
• youtube: https://www.youtube.com/watch?v=fE3FX8FowiU

Video Captioning and Retrieval Models with Semantic Attention

• intro: Winner of three (fill-in-the-blank, multiple-choice test, and movie retrieval) out of four tasks of the LSMDC 2016 Challenge (Workshop in ECCV 2016)
• arxiv: https://arxiv.org/abs/1610.02947

Spatio-Temporal Attention Models for Grounded Video Captioning

• arxiv: https://arxiv.org/abs/1610.04997

Video and Language: Bridging Video and Language with Deep Learning

• intro: ECCV-MM 2016. captioning, commenting, alignment
• slides: https://www.microsoft.com/en-us/research/wp-content/uploads/2016/10/Video-and-Language-ECCV-MM-2016-Tao-Mei-Pub.pdf

Recurrent Memory Addressing for describing videos

• arxiv: https://arxiv.org/abs/1611.06492

Video Captioning with Transferred Semantic Attributes

• arxiv: https://arxiv.org/abs/1611.07675

Adaptive Feature Abstraction for Translating Video to Language

• arxiv: https://arxiv.org/abs/1611.07837

Semantic Compositional Networks for Visual Captioning

• intro: CVPR 2017. Duke University & Tsinghua University & MSR
• arxiv: https://arxiv.org/abs/1611.08002
• github: https://github.com/zhegan27/SCN_for_video_captioning

Hierarchical Boundary-Aware Neural Encoder for Video Captioning

• arxiv: https://arxiv.org/abs/1611.09312

Attention-Based Multimodal Fusion for Video Description

• arxiv: https://arxiv.org/abs/1701.03126

Weakly Supervised Dense Video Captioning

• intro: CVPR 2017
• arxiv: https://arxiv.org/abs/1704.01502

Generating Descriptions with Grounded and Co-Referenced People

• intro: CVPR 2017. movie description
• arxiv: https://arxiv.org/abs/1704.01518

Multi-Task Video Captioning with Video and Entailment Generation

• intro: ACL 2017. UNC Chapel Hill
• arxiv: https://arxiv.org/abs/1704.07489

Dense-Captioning Events in Videos

• project page: http://cs.stanford.edu/people/ranjaykrishna/densevid/
• arxiv: https://arxiv.org/abs/1705.00754

Hierarchical LSTM with Adjusted Temporal Attention for Video Captioning

https://arxiv.org/abs/1706.01231

Reinforced Video Captioning with Entailment Rewards

• intro: EMNLP 2017. UNC Chapel Hill
• arxiv: https://arxiv.org/abs/1708.02300

End-to-end Concept Word Detection for Video Captioning, Retrieval, and Question Answering

• intro: CVPR 2017. Winner of three (fill-in-the-blank, multiple-choice test, and movie retrieval) out of four tasks of the LSMDC 2016 Challenge
• arxiv: https://arxiv.org/abs/1610.02947
• slides: https://drive.google.com/file/d/0B9nOObAFqKC9aHl2VWJVNFp1bFk/view

From Deterministic to Generative: Multi-Modal Stochastic RNNs for Video Captioning

https://arxiv.org/abs/1708.02478

Grounded Objects and Interactions for Video Captioning

https://arxiv.org/abs/1711.06354

Integrating both Visual and Audio Cues for Enhanced Video Caption

https://arxiv.org/abs/1711.08097

Video Captioning via Hierarchical Reinforcement Learning

https://arxiv.org/abs/1711.11135

Consensus-based Sequence Training for Video Captioning

https://arxiv.org/abs/1712.09532

Less Is More: Picking Informative Frames for Video Captioning

https://arxiv.org/abs/1803.01457

End-to-End Video Captioning with Multitask Reinforcement Learning

https://arxiv.org/abs/1803.07950

End-to-End Dense Video Captioning with Masked Transformer

• intro: CVPR 2018. University of Michigan & Salesforce Research
• arxiv: https://arxiv.org/abs/1804.00819

Reconstruction Network for Video Captioning

• intro: CVPR 2018
• arxiv: https://arxiv.org/abs/1803.11438

Bidirectional Attentive Fusion with Context Gating for Dense Video Captioning

• intro: CVPR 2018 spotlight paper
• arxiv: https://arxiv.org/abs/1804.00100

Jointly Localizing and Describing Events for Dense Video Captioning

• intro: CVPR 2018 Spotlight, Rank 1 in ActivityNet Captions Challenge 2017
• arxiv: https://arxiv.org/abs/1804.08274

Contextualize, Show and Tell: A Neural Visual Storyteller

https://arxiv.org/abs/1806.00738

RUC+CMU: System Report for Dense Captioning Events in Videos

• intro: Winner in ActivityNet 2018 Dense Video Captioning challenge
• arxiv: https://arxiv.org/abs/1806.08854

Streamlined Dense Video Captioning

• intro: CVPR 2019
• arxiv: https://arxiv.org/abs/1904.03870

Projects

Learning CNN-LSTM Architectures for Image Caption Generation: An implementation of CNN-LSTM image caption generator architecture that achieves close to state-of-the-art results on the MSCOCO dataset.

• github: https://github.com/mosessoh/CNN-LSTM-Caption-Generator

screengrab-caption: an openframeworks app that live-captions your desktop screen with a neural net

• intro: openframeworks app which grabs your desktop screen, then sends it to darknet for captioning. works great with video calls.
• github: https://github.com/genekogan/screengrab-caption

Tools

CaptionBot (Microsoft)

• website: https://www.captionbot.ai/

Blogs

Captioning Novel Objects in Images

http://bair.berkeley.edu/jacky/2017/08/08/novel-object-captioning/

Courses

(Toronto) CSC2541: Visual Perception for Autonomous Driving, Winter 2016

• homepage: http://www.cs.toronto.edu/~urtasun/courses/CSC2541/CSC2541_Winter16.html

(MIT) 6.S094: Deep Learning for Self-Driving Cars

• homepage: http://selfdrivingcars.mit.edu/
• github: https://github.com/lexfridman/deepcars
• youtube: https://www.youtube.com/playlist?list=PLrAXtmErZgOeiKm4sgNOknGvNjby9efdf
• mirror: https://pan.baidu.com/s/1boLRFaB

How to Land An Autonomous Vehicle Job: Coursework

• blog: https://medium.com/self-driving-cars/how-to-land-an-autonomous-vehicle-job-coursework-e7acc2bfe740#.7vfjx3i1j

Papers

An Empirical Evaluation of Deep Learning on Highway Driving

• arxiv: http://arxiv.org/abs/1504.01716
• github: https://github.com/brodyh/caffe

Real-time Joint Object Detection and Semantic Segmentation Network for Automated Driving

• intro: NeurIPS 2018 Workshop on Machine Learning on the Phone and other Consumer Devices (MLPCD 2)
• arxiv: https://arxiv.org/abs/1901.03912

Optical Flow augmented Semantic Segmentation networks for Automated Driving

• intro: VISAPP 2019 Oral
• arxiv: https://arxiv.org/abs/1901.07355

AuxNet: Auxiliary tasks enhanced Semantic Segmentation for Automated Driving

• intro: Short Paper for a poster presentation at VISAPP 2019
• arxiv: https://arxiv.org/abs/1901.05808

Design of Real-time Semantic Segmentation Decoder for Automated Driving

• intro: VISAPP 2019
• arxiv: https://arxiv.org/abs/1901.06580

Hierarchical Multi-task Deep Neural Network Architecture for End-to-End Driving

https://arxiv.org/abs/1902.03466

DeepDriving

DeepDriving: Learning Affordance for Direct Perception in Autonomous Driving

• project page: http://deepdriving.cs.princeton.edu/
• paper: http://deepdriving.cs.princeton.edu/paper.pdf
• code: http://deepdriving.cs.princeton.edu/DeepDriving.zip

End to End Learning for Self-Driving Cars

• intro: NVIDIA DevBox and Torch 7, 30 FPS
• arxiv: http://arxiv.org/abs/1604.07316
• blog: https://devblogs.nvidia.com/parallelforall/deep-learning-self-driving-cars/
• demo: https://www.youtube.com/watch?v=NJU9ULQUwng&feature=youtu.be
• github: https://github.com/SullyChen/Nvidia-Autopilot-TensorFlow

End-to-End Deep Learning for Self-Driving Cars

• blog: https://devblogs.nvidia.com/parallelforall/deep-learning-self-driving-cars/

Can we unify monocular detectors for autonomous driving by using the pixel-wise semantic segmentation of CNNs?

• arxiv: http://arxiv.org/abs/1607.00971

BRAIN4CARS: Cabin Sensing for Safe and Personalized Driving

Brain4Cars: Sensory-Fusion Recurrent Neural Models for Driver Activity Anticipation

Brain4Cars: Car That Knows Before You Do via Sensory-Fusion Deep Learning Architecture

• arxiv: http://arxiv.org/abs/1601.00740

Car that Knows Before You Do: Anticipating Maneuvers via Learning Temporal Driving Models

• arxiv: http://arxiv.org/abs/1504.02789
• github: https://github.com/asheshjain399/ICCV2015_Brain4Cars

Recurrent Neural Networks for Driver Activity Anticipation via Sensory-Fusion Architecture

• project page: http://www.brain4cars.com/
• arxiv: http://arxiv.org/abs/1509.05016
• github: https://github.com/asheshjain399/RNNexp

Long-term Planning by Short-term Prediction

• arxiv: http://arxiv.org/abs/1602.01580

Learning a Driving Simulator

• introo: by hacker Geohot
• project page: http://research.comma.ai/
• arxiv: http://arxiv.org/abs/1608.01230
• paper: https://github.com/commaai/research/blob/master/paper/commalds.pdf
• github: https://github.com/commaai/research

Comma.ai open-sources the data it used for its first successful driverless trips

• blog: https://techcrunch.com/2016/08/03/comma-ai-open-sources-the-data-it-used-for-its-first-successful-driverless-trips/

Autonomous driving challenge: To Infer the property of a dynamic object based on its motion pattern using recurrent neural network

• arxiv: http://arxiv.org/abs/1609.00361

Safe, Multi-Agent, Reinforcement Learning for Autonomous Driving

• arxiv: https://arxiv.org/abs/1610.03295

Learning from Maps: Visual Common Sense for Autonomous Driving

• arxiv: https://arxiv.org/abs/1611.08583

SAD-GAN: Synthetic Autonomous Driving using Generative Adversarial Networks

• intro: Accepted at the Deep Learning for Action and Interaction Workshop, 30th Conference on Neural Information Processing Systems (NIPS 2016)
• arxiv: https://arxiv.org/abs/1611.08788

MultiNet: Real-time Joint Semantic Reasoning for Autonomous Driving

• intro: first place on Kitti Road Segmentation. joint classification, detection and semantic segmentation via a unified architecture, less than 100 ms to perform all tasks
• arxiv: https://arxiv.org/abs/1612.07695
• github: https://github.com/MarvinTeichmann/MultiNet

Interpretable Learning for Self-Driving Cars by Visualizing Causal Attention

• intro: UC Berkeley
• arxiv: https://arxiv.org/abs/1703.10631

Virtual to Real Reinforcement Learning for Autonomous Driving

• intro: Shanghai Jiao Tong University & UC Berkeley & Tsinghua University
• arxiv: https://arxiv.org/abs/1704.03952

Computer Vision for Autonomous Vehicles: Problems, Datasets and State-of-the-Art

• homepage: http://www.cvlibs.net/projects/autonomous_vision_survey/
• arxiv: https://arxiv.org/abs/1704.05519

Deep Reinforcement Learning framework for Autonomous Driving

https://arxiv.org/abs/1704.02532

Systematic Testing of Convolutional Neural Networks for Autonomous Driving

https://arxiv.org/abs/1708.03309

MODNet: Moving Object Detection Network with Motion and Appearance for Autonomous Driving

https://arxiv.org/abs/1709.04821

CFENet: An Accurate and Efficient Single-Shot Object Detector for Autonomous Driving

• intro: CVPR 2018 Workshop of Autonomous Driving (WAD)
• arxiv: https://arxiv.org/abs/1806.09790

LaneNet: Real-Time Lane Detection Networks for Autonomous Driving

• intro: Duke University & Horizon Robotics, Inc.
• arxiv: https://arxiv.org/abs/1807.01726

Learning End-to-end Autonomous Driving using Guided Auxiliary Supervision

https://arxiv.org/abs/1808.10393

Rethinking Self-driving: Multi-task Knowledge for Better Generalization and Accident Explanation Ability

• intro: Waseda University
• arxiv: https://arxiv.org/abs/1809.11100
• demo: https://www.youtube.com/watch?v=N7ePnnZZwdE

Pixel and Feature Level Based Domain Adaption for Object Detection in Autonomous Driving

https://arxiv.org/abs/1810.00345

Multi-task Learning with Attention for End-to-end Autonomous Driving

• intro: CVPR 2021 Workshop on Autonomous Driving
• arxiv: https://arxiv.org/abs/2104.10753

MP3: A Unified Model to Map, Perceive, Predict and Plan

• intro: Uber ATG & University of Toronto
• arxiv: https://arxiv.org/abs/2101.06806

Level 2 Autonomous Driving on a Single Device: Diving into the Devils of Openpilot

• intro: Shanghai AI Laboratory & Shanghai Jiao Tong University & UCSD & SenseTime
• arxiv: https://arxiv.org/abs/2206.08176
• github: https://github.com/OpenPerceptionX/Openpilot-Deepdive

Real-time Full-stack Traffic Scene Perception for Autonomous Driving with Roadside Cameras

• intro: ICRA 2022
• intro: University of Michigan & Ford Motor Company
• arxiv: https://arxiv.org/abs/2206.09770

ST-P3: End-to-end Vision-based Autonomous Driving via Spatial-Temporal Feature Learning

• intro: ECCV 2022
• intro: Shanghai Jiao Tong University & Shanghai AI Laboratory & The University of California & JD Explore Academy
• arxiv: https://arxiv.org/abs/2207.07601
• github: https://github.com/OpenPerceptionX/ST-P3

Effective Adaptation in Multi-Task Co-Training for Unified Autonomous Driving

• intro: NeurIPS 2022
• intro: Shenzhen Campus of Sun Yat-Sen University & Huawei Noah’s Ark Lab
• arxiv: https://arxiv.org/abs/2209.08953

Planning-oriented Autonomous Driving

• intro: CVPR 2023 best paper
• intro: Shanghai AI Laboratory, Wuhan University, SenseTime Research
• project page: https://opendrivelab.github.io/UniAD/
• arxiv: https://arxiv.org/abs/2212.10156
• github: https://github.com/OpenDriveLab/UniAD

Projects

Caffe-Autopilot: Car autopilot software that uses C++, BVLC Caffe, OpenCV, and SFML

• github: https://github.com/SullyChen/Caffe-Autopilot

Self Driving Car Demo

• intro; A project that trains a virtual car to how to move an object around a screen (drive itself) without running into obstacles using a type of reinforcement learning called Q-Learning
• github: https://github.com/llSourcell/Self-Driving-Car-Demo/

Autoware: Open-source software for urban autonomous driving

• github: https://github.com/CPFL/Autoware

Open Sourcing 223GB of Driving Data

• homepage: https://udacity.com/self-driving-car
• blog: https://medium.com/udacity/open-sourcing-223gb-of-mountain-view-driving-data-f6b5593fbfa5#.q8nk5bfpp
• github: https://github.com/udacity/self-driving-car

Machine Learning for RC Cars

• github: https://github.com/kendricktan/suiron

Self Driving (Toy) Ferrari

• github: https://github.com/RyanZotti/Self-Driving-Car

Lane Finding Project for Self-Driving Car ND

• github: https://github.com/udacity/CarND-LaneLines-P1

Instructions on how to get your development environment ready for Udacity Self Driving Car (SDC) Challenges

• github: https://github.com/gtarobotics/self-driving-car

DeepDrive: self-driving car AI

• intro: Caffe Model / Dataset / Tips and Tricks
• homepage: http://deepdrive.io/

DeepDrive setup: Run a self-driving car simulator from the comfort of your own PC

• github: https://github.com/crizCraig/deepdrive

DeepTesla: End-to-End Learning from Human and Autopilot Driving

http://selfdrivingcars.mit.edu/deeptesla/

DeepPicar: A Low-cost Deep Neural Network-based Autonomous Car

• arxiv: https://arxiv.org/abs/1712.08644
• github: https://github.com//heechul/picar

Autonomous Driving in Reality with Reinforcement Learning and Image Translation

• intro: Shanghai Jiao Tong University
• arxiv: https://arxiv.org/abs/1801.05299

End-to-end Multi-Modal Multi-Task Vehicle Control for Self-Driving Cars with Visual Perception

https://arxiv.org/abs/1801.06734

Blogs

Self-driving cars: How far away are we REALLY from autonomous cars?(7 Aug 2015)

http://www.alphr.com/cars/1001329/self-driving-cars-how-far-away-are-we-really-from-autonomous-cars

Practice makes perfect: Driverless cars will learn from their mistakes(9 Oct 2015)

http://www.alphr.com/cars/1001713/practice-makes-perfect-driverless-cars-will-learn-from-their-mistakes

Eyes on the Road: How Autonomous Cars Understand What They’re Seeing

• blog: http://blogs.nvidia.com/blog/2016/01/05/eyes-on-the-road-how-autonomous-cars-understand-what-theyre-seeing/

Human-in-the-loop deep learning will help drive autonomous cars

http://venturebeat.com/2016/06/25/human-in-the-loop-deep-learning-will-help-drive-autonomous-cars/

Using reinforcement learning in Python to teach a virtual car to avoid obstacles

• part 1: https://medium.com/@harvitronix/using-reinforcement-learning-in-python-to-teach-a-virtual-car-to-avoid-obstacles-6e782cc7d4c6#.rneyuerga
• part 2: https://medium.com/@harvitronix/reinforcement-learning-in-python-to-teach-a-virtual-car-to-avoid-obstacles-part-2-93e614fcd238#.1pt1lli4c
• part 3: https://medium.com/@harvitronix/reinforcement-learning-in-python-to-teach-an-rc-car-to-avoid-obstacles-part-3-a1d063ac962f#.jwzm2v1r4
• github: https://github.com/harvitronix/reinforcement-learning-car

Autonomous RC car using Raspberry Pi and Neural Networks

• blog: http://www.multunus.com/blog/2016/07/autonomous-rc-car-using-raspberry-pi-and-neural-networks/
• github: https://github.com/multunus/autonomous-rc-car

The Road Ahead: Autonomous Vehicles Startup Ecosystem

https://medium.com/the-mission/the-road-ahead-autonomous-vehicles-startup-ecosystem-3c91d546673d#.gft1xyh9l

Deep Driving - A revolutionary AI technique is about to transform the self-driving car

https://www.technologyreview.com/s/602600/deep-driving/

Visualizations for regressing wheel steering angles in self driving cars with Keras

• blog: http://jacobcv.blogspot.jp/2016/10/visualizations-for-regressing-wheel.html
• github: https://github.com/jacobgil/keras-steering-angle-visualizations