Deep Learning Tricks

Papers

Practical recommendations for gradient-based training of deep architectures

• author: Yoshua Bengio
• arxiv: http://arxiv.org/abs/1206.5533

Bag of Tricks for Image Classification with Convolutional Neural Networks

• intro: Amazon Web Services
• arxiv: https://arxiv.org/abs/1812.01187

Blogs

Efficient BackProp

• intro: Neural Networks: Tricks of the Trade, 2nd
• blog: http://blog.csdn.net/zouxy09/article/details/45288129

Deep Learning for Vision: Tricks of the Trade

• intro: CVPR. Marc’Aurelio Ranzato
• slides: http://bavm2013.splashthat.com/img/events/46439/assets/34a7.ranzato.pdf

Optimizing RNN performance

• intro: Silicon Valley AI Lab
• keywords: Optimize GEMM, parallel GPU, GRU and LSTM…
• blog: http://svail.github.io/

Must Know Tips/Tricks in Deep Neural Networks

• intro: by Xiu-Shen Wei, NJU LAMDA
• blog: http://lamda.nju.edu.cn/weixs/project/CNNTricks/CNNTricks.html
• slides: http://lamda.nju.edu.cn/weixs/slide/CNNTricks_slide.pdf

Training Tricks from Deeplearning4j

http://deeplearning4j.org/trainingtricks.html

Suggestions for DL from Llya Sutskeve

• intro: data, preprocessing, mini-batch, gradient normalization, learning rate, weight initialization, data augmentation, dropout and ensemble
• blog: http://yyue.blogspot.com/2015/01/a-brief-overview-of-deep-learning.html

Efficient Training Strategies for Deep Neural Network Language Models

• intro: batch-size, initial learning rate, network initialization
• blog: https://fb56552f-a-62cb3a1a-s-sites.googlegroups.com/site/deeplearningworkshopnips2014/71.pdf?attachauth=ANoY7cp_eDwTXPm6iWHdBRhlIsgPASEAwkW-exLSOsz467mge7zLCkBMWznOu_G90vGVtqNvXOusc4z6cC6hEnHk6YzHtuEr_kyU0fyme7asaECN0zvoNwDk5258CueoB6fY3WtLvbJzYok1xiIeWSFYtk5mKXCXFDMI6djwhjCX1xi0GEEv_x7uMQwTdQlDItZ3kgLnZ2RjctQmIXDCu58fS3Wby4vWX3CkhMIf_EpCXx7jDn_M2SM%3D&attredirects=0

Neural Networks Best Practice

• intro: Uber
• paper: http://www.kentran.net/2013/04/neural-network-best-practices.html

Dark Knowledge from Hinton

• youtube: https://www.youtube.com/watch?v=EK61htlw8hY
• slides: http://www.ttic.edu/dl/dark14.pdf
• notes: http://deepdish.io/2014/10/28/hintons-dark-knowledge/
• notes: http://fastml.com/geoff-hintons-dark-knowledge/

Stochastic Gradient Descent Tricks(Leon Bottou)

http://leon.bottou.org/publications/pdf/tricks-2012.pdf

Advice for applying Machine Learning

https://jmetzen.github.io/2015-01-29/ml_advice.html

How to Debug Learning Algorithm for Regression Model

http://vitalflux.com/machine-learning-debug-learning-algorithm-regression-model/

Large-scale L-BFGS using MapReduce

• intro: NIPS 2014
• paper: http://papers.nips.cc/paper/5333-large-scale-l-bfgs-using-mapreduce.pdf

Selecting good features

– Part I: univariate selection: http://blog.datadive.net/selecting-good-features-part-i-univariate-selection/ – Part II: linear models and regularization: http://blog.datadive.net/selecting-good-features-part-ii-linear-models-and-regularization/ – Part III: random forests: http://blog.datadive.net/selecting-good-features-part-iii-random-forests/ – Part IV: stability selection, RFE and everything side by side: http://blog.datadive.net/selecting-good-features-part-iv-stability-selection-rfe-and-everything-side-by-side/

机器学习代码心得之有监督学习的模块

http://www.weibo.com/p/1001603795687165852957

Stochastic Gradient Boosting: Choosing the Best Number of Iterations

• intro: Kaggle winner YANIR SEROUSSI
• blog: http://yanirseroussi.com/2014/12/29/stochastic-gradient-boosting-choosing-the-best-number-of-iterations/

Large-Scale High-Precision Topic Modeling on Twitter

• intro: Twitter senior researcher. KDD 2014
• paper: http://www.eeshyang.com/papers/KDD14Jubjub.pdf

H2O World - Top 10 Deep Learning Tips & Tricks - Arno Candel

http://www.slideshare.net/0xdata/h2o-world-top-10-deep-learning-tips-tricks-arno-candel

How To Improve Deep Learning Performance: 20 Tips, Tricks and Techniques That You Can Use To Fight Overfitting and Get Better Generalization

http://machinelearningmastery.com/improve-deep-learning-performance/

Neural Network Training Speed Trick

https://medium.com/machine-learning-at-petiteprogrammer/neural-network-training-speed-trick-92d6b22a7754#.4v6qukpn7

The Black Magic of Deep Learning - Tips and Tricks for the practitioner

http://nmarkou.blogspot.ru/2017/02/the-black-magic-of-deep-learning-tips.html