Deep Learning Software and Hardware

Papers

Accelerating Deep Convolutional Neural Networks Using Specialized Hardware

• paper: http://research.microsoft.com/pubs/240715/CNN%20Whitepaper.pdf

Installation / Deploying

Setting up a Deep Learning Machine from Scratch (Software): Instructions for setting up the software on your deep learning machine

• intro: A detailed guide to setting up your machine for deep learning research. Includes instructions to install drivers, tools and various deep learning frameworks. This was tested on a 64 bit machine with Nvidia Titan X, running Ubuntu 14.04
• github: https://github.com/saiprashanths/dl-setup

How to install CUDA Toolkit and cuDNN for deep learning

• blog: http://www.pyimagesearch.com/2016/07/04/how-to-install-cuda-toolkit-and-cudnn-for-deep-learning/

Deploying Deep Learning: Guide to deploying deep-learning inference networks and realtime object detection with TensorRT and Jetson TX1.

• github: https://github.com/dusty-nv/jetson-inference

Install Log

• intro: setting up Caffe on a cluster running Redhat 6.3 (Santiago) without having root
• github: https://github.com/yosinski/caffe/blob/jason_public/doc/linux-no-root-install-log.md

Lessons Learned from Deploying Deep Learning at Scale

• blog: http://blog.algorithmia.com/deploying-deep-learning-cloud-services/

Docker

All-in-one Docker image for Deep Learning

• intro: An all-in-one Docker image for deep learning. Contains all the popular DL frameworks (TensorFlow, Theano, Torch, Caffe, etc.)
• github: https://github.com/saiprashanths/dl-docker

NVIDIA Docker: GPU Server Application Deployment Made Easy

• blog: https://devblogs.nvidia.com/parallelforall/nvidia-docker-gpu-server-application-deployment-made-easy/
• github: https://github.com/NVIDIA/nvidia-docker

Deep learning base image for Docker (Tensorflow, Caffe, MXNet, Torch, Openface, etc.)

https://github.com/dominiek/deep-base

Deepo: a Docker image with a full reproducible deep learning research environment

• intro: A Docker image containing almost all popular deep learning frameworks: theano, tensorflow, sonnet, pytorch, keras, lasagne, mxnet, cntk, chainer, caffe, torch.
• project page: https://hub.docker.com/r/ufoym/deepo/
• github: https://github.com//ufoym/deepo

Cloud

SuperVessel Cloud for POWER/OpenPOWER LoginRegisterTutorials

http://www.ptopenlab.com/

Building Deep Neural Networks in the Cloud with Azure GPU VMs, MXNet and Microsoft R Server

https://blogs.technet.microsoft.com/machinelearning/2016/09/15/building-deep-neural-networks-in-the-cloud-with-azure-gpu-vms-mxnet-and-microsoft-r-server/

Microsoft open sources its next-gen cloud hardware design

• blog: https://techcrunch.com/2016/10/31/microsoft-open-sources-its-next-gen-cloud-hardware-design/

Google Taps AMD For Accelerating Machine Learning In The Cloud

http://www.forbes.com/sites/aarontilley/2016/11/15/google-taps-amd-for-accelerating-machine-learning-in-the-cloud/#3549d8554181

Amazon EC2

Deep Learning AMI on AWS Marketplace

https://aws.amazon.com/marketplace/pp/B01M0AXXQB

We Have To Go Deeper: AWS p2.xlarge GPU optimized deep learning cluster-grenade

• github: https://github.com/Miej/GoDeeper

A GPU enabled AMI for Deep Learning

• blog: https://blog.empiricalci.com/a-gpu-enabled-ami-for-deep-learning-5aa3d694b630#.9339zxm4e

Keras with GPU on Amazon EC2 – a step-by-step instruction

https://medium.com/@mateuszsieniawski/keras-with-gpu-on-amazon-ec2-a-step-by-step-instruction-4f90364e49ac#.k27d0mqir

Microsoft R Server

Training Deep Neural Networks on ImageNet Using Microsoft R Server and Azure GPU VMs

• blog: https://blogs.technet.microsoft.com/machinelearning/2016/11/15/imagenet-deep-neural-network-training-using-microsoft-r-server-and-azure-gpu-vms/

Hardware System

I: Building a Deep Learning (Dream) Machine

• blog: http://graphific.github.io/posts/building-a-deep-learning-dream-machine/
• slides: http://www.slideshare.net/roelofp/building-a-deep-learning-dream-machine

II: Running a Deep Learning (Dream) Machine

• blog: http://graphific.github.io/posts/running-a-deep-learning-dream-machine/

A Full Hardware Guide to Deep Learning

• blog: http://timdettmers.com/2015/03/09/deep-learning-hardware-guide/

Build your own Deep Learning Box

• blog: https://annalyzin.wordpress.com/2016/05/19/build-a-deep-learning-box/

32-TFLOP Deep Learning GPU Box: A super-fast linux-based machine with multiple GPUs for training deep neural nets

https://hackaday.io/project/12070-32-tflop-deep-learning-gpu-box

Hands-on with the NVIDIA DIGITS DevBox for Deep Learning

• blog: http://www.pyimagesearch.com/2016/06/06/hands-on-with-the-nvidia-digits-devbox-for-deep-learning/

Considerations when setting up deep learning hardware

• blog: http://www.pyimagesearch.com/2016/06/13/considerations-when-setting-up-deep-learning-hardware/

Building a Workstation for Deep Learning

• slides: http://www.slideshare.net/PetteriTeikariPhD/deep-learning-workstation

Deep Learning Machine: First build experience

• blog: https://medium.com/@vivek.yadav/deep-learning-machine-first-build-experience-d04abf198831#.1d6q5mw9m

Building a machine learning/deep learning workstation for under $5000

• blog: https://www.analyticsvidhya.com/blog/2016/11/building-a-machine-learning-deep-learning-workstation-for-under-5000/

Hardware Guide: Neural Networks on GPUs (Updated 2016-1-30)

• intro: by Joseph Redmon
• blog: http://pjreddie.com/darknet/hardware-guide/

Building Your Own Deep Learning Box

https://medium.com/@bfortuner/building-your-own-deep-learning-box-47b918aea1eb#.4r5zchk4f

Setting up a Deep learning machine in a lazy yet quick way https://medium.com/@sravsatuluri/setting-up-a-deep-learning-machine-in-a-lazy-yet-quick-way-be2642318850#.jrxrkfxa2

Deep Confusion: Misadventures In Building A Deep Learning Machine

http://www.topbots.com/deep-confusion-misadventures-in-building-a-machine-learning-server/

DIY-Deep-Learning-Workstation

• intro: Build a deep learning workstation from scratch (HW & SW).
• github: https://github.com/charlesq34/DIY-Deep-Learning-Workstation

GPU

Which GPU(s) to Get for Deep Learning: My Experience and Advice for Using GPUs in Deep Learning

• blog: http://timdettmers.com/2017/04/09/which-gpu-for-deep-learning/

从深度学习选择什么样的gpu来谈谈gpu的硬件架构

• blog: http://chenrudan.github.io/blog/2015/12/20/introductionofgpuhardware.html

GPU折腾手记——2015 (by 李沐)

• blog: http://mli.github.io/gpu/2016/01/17/build-gpu-clusters/

HPC, Deep Learning and GPUs(2016 Stanford HPC Conference)

• youtube: https://www.youtube.com/watch?v=JwgoC-1V_38
• video: http://pan.baidu.com/s/1pKrSvOZ

Modern GPU 2.0: Design patterns for GPU computing

• intro: Modern GPU is code and commentary intended to promote new and productive ways of thinking about GPU computing.
• homepage: http://nvlabs.github.io/moderngpu/
• github: https://github.com/nvlabs/moderngpu

CuMF: CUDA-Acclerated ALS on mulitple GPUs.

• github: https://github.com/wei-tan/CuMF

Basic Performance Analysis of NVIDIA GPU Accelerator Cards for Deep Learning Applications

• wihte paper: https://www.amax.com/enterprise/pdfs/Deep%20Learning%20Performance%20Analysis.pdf

CuPy : NumPy-like API accelerated with CUDA

• github: https://github.com/pfnet/cupy

NumPy GPU acceleration

• blog: http://scottsievert.com/blog/2016/07/01/numpy-gpu/

Efficient Convolutional Neural Network Inference on Mobile GPUs (Embedded Vision Summit)

• youtube: https://www.youtube.com/watch?v=ximyhmm17UM

Deep Learning with Multiple GPUs on Rescale: Torch

• blog: https://blog.rescale.com/deep-learning-with-multiple-gpus-on-rescale-torch/

GPU-accelerated Theano & Keras on Windows 10 native

• arxiv: https://github.com/philferriere/dlwin

NVIDIA Announces Quadro GP100 - Big Pascal Comes to Workstations

http://www.anandtech.com/show/11102/nvidia-announces-quadro-gp100

FPGA

Recurrent Neural Networks Hardware Implementation on FPGA

• arxiv: http://arxiv.org/abs/1511.05552

Is implementing deep learning on FPGAs a natural next step after the success with GPUs?

• quora: https://www.quora.com/Is-implementing-deep-learning-on-FPGAs-a-natural-next-step-after-the-success-with-GPUs

Efficient Implementation of Neural Network Systems Built on FPGAs, Programmed with OpenCL

• paper: https://www.altera.com/content/dam/altera-www/global/en_US/pdfs/literature/solution-sheets/efficient_neural_networks.pdf?utm_source=Altera&utm_medium=link&utm_campaign=OpenCL_15_1&utm_content=NA_efficient-neural-networks-solution-sheet-download-link

Deep Learning on FPGAs: Past, Present, and Future

• arxiv: http://arxiv.org/abs/1602.04283

FPGAs Challenge GPUs as a Platform for Deep Learning

• blog: https://www.tractica.com/automation-robotics/fpgas-challenge-gpus-as-a-platform-for-deep-learning/

Convolution Neural Network CNN Implementation on Altera FPGA using OpenCL

• youtube: https://www.youtube.com/watch?v=78Qd5t-Mn0s

Accelerating Deep Learning Using Altera FPGAs (Embedded Vision Summit)

• youtube: https://www.youtube.com/watch?v=HlBC9qBqZRs
• slides: http://www.slideshare.net/embeddedvision/accelerating-deep-learning-using-altera-fpgas-a-presentation-from-intel

Machine Learning on FPGAs: Neural Networks

• youtube: https://www.youtube.com/watch?v=3iCifD8gZ0Q

Comprehensive Evaluation of OpenCL-based Convolutional Neural Network Accelerators in Xilinx and Altera FPGAs

• arxiv: https://arxiv.org/abs/1609.09296

Microsoft Goes All in for FPGAs to Build Out AI Cloud

• blog: https://www.top500.org/news/microsoft-goes-all-in-for-fpgas-to-build-out-cloud-based-ai/

Caffeinated FPGAs: FPGA Framework For Convolutional Neural Networks

• arxiv: https://arxiv.org/abs/1609.09671
• github: https://github.com/dicecco1/fpga_caffe

Intel Unveils FPGA to Accelerate Neural Networks

http://datacenterfrontier.com/intel-unveils-fpga-to-accelerate-ai-neural-networks/

Deep Learning with FPGA

• blog: https://amundtveit.com/2016/11/23/deep-learning-with-fpga/

A General Neural Network Hardware Architecture on FPGA

• intro: University of Birmingham
• arxiv: https://arxiv.org/abs/1711.05860

Approximate FPGA-based LSTMs under Computation Time Constraints

• intro: ARC 2018
• arxiv: https://arxiv.org/abs/1801.02190

ARM / Processor

‘Neural network’ spotted deep inside Samsung’s Galaxy S7 silicon brain: Secrets of Exynos M1 cores spilled

• blog: http://www.theregister.co.uk/2016/08/22/samsung_m1_core/?mt=1471918256061

Intel will add deep-learning instructions to its processors

• blog: http://lemire.me/blog/2016/10/14/intel-will-add-deep-learning-instructions-to-its-processors/

SRAM

ShiDianNao: Shifting Vision Processing Closer to the Sensor http://lap.epfl.ch/files/content/sites/lap/files/shared/publications/DuJun15_ShiDianNaoShiftingVisionProcessingCloserToTheSensor_ISCA15.pdf

Blogs

Emerging “Universal” FPGA, GPU Platform for Deep Learning

• blog: http://www.nextplatform.com/2016/06/29/universal-fpga-gpu-platform-deep-learning/

An Early Look at Startup Graphcore’s Deep Learning Chip

https://www.nextplatform.com/2017/03/09/early-look-startup-graphcores-deep-learning-chip/

Hardware for Deep Learning

https://medium.com/towards-data-science/hardware-for-deep-learning-8d9b03df41a

Videos

Energy-efficient Hardware for Embedded Vision and Deep Convolutional Neural Networks

• intro: September 2016 Embedded Vision Alliance Member Meeting Presentation: MIT
• youtube: https://www.youtube.com/watch?v=dO_lHz87DVM