Fun With Deep Learning

Painting

Artist Agent: A Reinforcement Learning Approach to Automatic Stroke Generation in Oriental Ink Painting

• intro: ICML 2012
• arxiv: https://arxiv.org/abs/1206.4634

Emoji

Brewing EmojiNet

• blog: http://engineering.curalate.com/2016/01/20/emojinet.html
• website: https://emojini.curalate.com/

Image2Emoji: Zero-shot Emoji Prediction for Visual Media

• paper: http://isis-data.science.uva.nl/cgmsnoek/pub/cappallo-image2emoji-mm2015.pdf

Teaching Robots to Feel: Emoji & Deep Learning 👾 💭 💕

• blog: http://getdango.com/emoji-and-deep-learning.html
• app: https://play.google.com/store/apps/details?id=co.dango.emoji.gif

Text input with relevant emoji sorted with deeplearning

• homepage: http://codepen.io/Idlework/pen/xOgGqM

Sketch

Sketch-a-Net that Beats Humans

• project page: http://www.eecs.qmul.ac.uk/~tmh/downloads.html
• arxiv: http://arxiv.org/abs/1501.07873
• paper: http://www.eecs.qmul.ac.uk/~tmh/papers/yu2015sketchanet.pdf
• code: http://www.eecs.qmul.ac.uk/~tmh/downloads/SketchANet_Code.zip

How Do Humans Sketch Objects?

• project page: http://cybertron.cg.tu-berlin.de/eitz/projects/classifysketch/
• paper: http://cybertron.cg.tu-berlin.de/eitz/pdf/2012_siggraph_classifysketch.pdf
• github: https://github.com/Zebreu/SketchingAI
• gitxiv: http://gitxiv.com/posts/ZBCxEc9g3Fg5xCQ6n/sketchingai

Learning to Simplify: Fully Convolutional Networks for Rough Sketch Cleanup (SIGGRAPH 2016)

• homepage: http://hi.cs.waseda.ac.jp/~esimo/en/research/sketch/
• paper: http://hi.cs.waseda.ac.jp/~esimo/publications/SimoSerraSIGGRAPH2016.pdf

Convolutional Sketch Inversion

• arxiv: http://arxiv.org/abs/1606.03073
• review: https://www.technologyreview.com/s/601684/machine-vision-algorithm-learns-to-transform-hand-drawn-sketches-into-photorealistic-images/
• review: https://techcrunch.com/2016/07/24/researchers-use-neural-networks-to-turn-face-sketches-into-photos/

Sketch Me That Shoe (CVPR 2016)

• project page: http://www.eecs.qmul.ac.uk/~qian/Project_cvpr16.html
• paper: http://www.eecs.qmul.ac.uk/~qian/SketchMeThatShoe.pdf
• github: https://github.com/seuliufeng/DeepSBIR

Mastering Sketching: Adversarial Augmentation for Structured Prediction

• keywords: Sketch Simplification
• project page: http://hi.cs.waseda.ac.jp/~esimo/en/research/sketch_master/
• arxiv: https://arxiv.org/abs/1703.08966
• github: https://github.com/bobbens/sketch_simplification

SketchyGAN: Towards Diverse and Realistic Sketch to Image Synthesis

• intro: Georgia Institute of Technology
• arxiv: https://arxiv.org/abs/1801.02753

Image Colorization

Deep Colorization

• paper: http://www.cs.cityu.edu.hk/~qiyang/publications/iccv-15.pdf

Learning Large-Scale Automatic Image Colorization

• paper: http://www.cv-foundation.org/openaccess/content_iccv_2015/papers/Deshpande_Learning_Large-Scale_Automatic_ICCV_2015_paper.pdf

Learning Representations for Automatic Colorization

• homepage: http://people.cs.uchicago.edu/~larsson/colorization/
• arxiv: http://arxiv.org/abs/1603.06668
• github: https://github.com/gustavla/autocolorize

Colorful Image Colorization

• intro: ECCV 2016
• project page: http://richzhang.github.io/colorization/
• arxiv: http://arxiv.org/abs/1603.08511
• github: https://github.com/richzhang/colorization
• demo: http://demos.algorithmia.com/colorize-photos/
• github: https://github.com/tdeboissiere/DeepLearningImplementations/tree/master/Colorful github(Tensorflow): https://github.com/nilboy/colorization-tf

Colorising Black & White Photos using Deep Learning

https://hackernoon.com/colorising-black-white-photos-using-deep-learning-4da22a05f531

---

Automatic Colorization (Tensorflow + VGG)

• blog: http://tinyclouds.org/colorize/

colornet: Neural Network to colorize grayscale images

https://github.com/pavelgonchar/colornet

Let there be Color!: Joint End-to-end Learning of Global and Local Image Priors for Automatic Image Colorization with Simultaneous Classification (SIGGRAPH 2016)

• homepage: http://hi.cs.waseda.ac.jp/~iizuka/projects/colorization/
• paper: http://hi.cs.waseda.ac.jp/~iizuka/projects/colorization/data/colorization_sig2016.pdf
• github(Torch7): https://github.com/satoshiiizuka/siggraph2016_colorization

Convolutional autoencoder to colorize greyscale images

• github: https://github.com/YerevaNN/neural-colorizer

Image-Color: A deep learning approach to colorizing images

• github: https://github.com/cameronfabbri/Colorful-Image-Colorization

Creating an artificial artist: Color your photos using Neural Networks

• blog: https://www.analyticsvidhya.com/blog/2016/11/creating-an-artificial-artist-color-your-photos-using-neural-networks/

Paints Chainer: line drawing colorization using chainer

• github: https://github.com/pfnet/PaintsChainer
• demo: http://paintschainer.preferred.tech/

Unsupervised Diverse Colorization via Generative Adversarial Networks

• arxiv: https://arxiv.org/abs/1702.06674

(DE)^2 CO: Deep Depth Colorization

https://arxiv.org/abs/1703.10881

A Neural Representation of Sketch Drawings

• intro: Google Brain
• arxiv: https://arxiv.org/abs/1704.03477

Real-Time User-Guided Image Colorization with Learned Deep Priors

• intro: SIGGRAPH 2017
• project page: https://richzhang.github.io/ideepcolor/
• arxiv: https://arxiv.org/abs/1705.02999
• github(official, Caffe): https://github.com/junyanz/interactive-deep-colorization

PixColor: Pixel Recursive Colorization

• intro: Google Research
• arxiv: https://arxiv.org/abs/1705.07208

cGAN-based Manga Colorization Using a Single Training Image

• intro: University of Tokyo
• arxiv: https://arxiv.org/abs/1706.06918

Interactive Deep Colorization With Simultaneous Global and Local Inputs

https://arxiv.org/abs/1801.09083

Image Colorization with Generative Adversarial Networks

https://arxiv.org/abs/1803.05400

Learning to Color from Language

• intro: Allen Institute of Artificial Intelligence & University of Massachusetts
• arxiv: https://arxiv.org/abs/1804.06026

Deep Exemplar-based Colorization

• intro: Siggraph 2018
• arxiv: https://arxiv.org/abs/1807.06587

Pixel-level Semantics Guided Image Colorization

https://arxiv.org/abs/1808.01597

User-Guided Deep Anime Line Art Colorization with Conditional Adversarial Networks

• intro: 2018 ACM Multimedia Conference (MM ‘18)
• arxiv: https://arxiv.org/abs/1808.03240

Pixelated Semantic Colorization

https://arxiv.org/abs/1901.10889

Colorization Transformer

• intro: ICLR 2021
• intro: Google Research, Brain Team
• arxiv: https://arxiv.org/abs/2102.04432
• openreview: https://openreview.net/forum?id=5NA1PinlGFu
• github: https://github.com/google-research/google-research/tree/master/coltran

Sounds

Visually Indicated Sounds

• project page: http://vis.csail.mit.edu/
• arxiv: http://arxiv.org/abs/1512.08512

Music

GRUV: Algorithmic Music Generation using Recurrent Neural Networks

• github: https://github.com/MattVitelli/GRUV

DeepHear - Composing and harmonizing music with neural networks

• website: http://web.mit.edu/felixsun/www/neural-music.html
• github: https://github.com/fephsun/neuralnetmusic

Using AutoHarp and a Character-Based RNN to Create MIDI Drum Loops

• website: http://www.inquisitivists.com/2015/09/16/using-autoharp-and-a-character-based-rnn-to-create-midi-drum-loops

Musical Audio Synthesis Using Autoencoding Neural Nets

• paper: http://www.cs.dartmouth.edu/~sarroff/papers/sarroff2014a.pdf
• github: https://github.com/woodshop/deepAutoController/tree/icmc_smc_2014
• video: https://vimeo.com/121827215

sound-rnn: Generating sound using recurrent neural networks

• github: https://github.com/johnglover/sound-rnn
• blog: http://www.johnglover.net/blog/generating-sound-with-rnns.html

Using LSTM Recurrent Neural Networks for Music Generation (Project for AI Prac Fall 2015 at Cornell)

• youtube: https://www.youtube.com/watch?v=aSr8_QQYpYM
• video: http://video.weibo.com/show?fid=1034:4be01d679bb1a68a634fe0f589caa779

Visually Indicated Sounds (MIT. 2015)

• arxiv: http://arxiv.org/abs/1512.08512

Training a Recurrent Neural Network to Compose Music

• blog: https://maraoz.com/2016/02/02/abc-rnn/

LSTM Realbook

• blog: https://keunwoochoi.wordpress.com/2016/02/19/lstm-realbook/
• github: https://github.com/keunwoochoi/lstm_real_book

LSTMetallica: Generation drum tracks by learning the drum tracks of 60 Metallica songs

• blog: https://keunwoochoi.wordpress.com/2016/02/23/lstmetallica/

deepjazz: Deep learning driven jazz generation using Keras & Theano!

• homepage: https://jisungk.github.io/deepjazz/
• github：https://github.com/jisungk/deepjazz

Magenta: Music and Art Generation with Machine Intelligence

• homepage: http://magenta.tensorflow.org/
• github: https://github.com/tensorflow/magenta

Music Transcription with Convolutional Neural Networks

• blog: https://www.lunaverus.com/cnn
• download: https://www.lunaverus.com/download

Long Short-Term Memory Recurrent Neural Network Architectures for Generating Music and Japanese Lyrics

• paper: http://cslab1.bc.edu/~csacademics/pdf/16Mikami.pdf
• github: https://github.com/mikamia

BachBot: Use deep learning to generate and harmonize music in the style of Bach

• intro: BachBot is a research project utilizing long short term memory (LSTMs) to generate Bach compositions
• homepage: http://bachbot.com/
• github: https://github.com/feynmanliang/bachbot

Generate Music in TensorFlow

• youtube: https://www.youtube.com/watch?v=ZE7qWXX05T0
• github: https://github.com/llSourcell/Music_Generator_Demo

Generate new lyrics in the style of any artist using LSTMs and TensorFlow

• github: https://github.com/dyelax/encore.ai

sound-GAN: Generative Adversial Network for music composition

• github: https://github.com/jacotar/sound-GAN

Analyzing Six Deep Learning Tools for Music Generation

• intro: Magenta / DeepJazz / BachBot / FlowMachines / WaveNet / GRUV
• blog: http://www.asimovinstitute.org/notes-vs-waves/

WIMP2: Creating Music with AI: Highlights of Current Research

• youtube: https://www.youtube.com/watch?v=5K4hn6cBUPU

Song From PI: A Musically Plausible Network for Pop Music Generation

• paper: http://openreview.net/pdf?id=ByBwSPcex
• project page: http://www.cs.toronto.edu/songfrompi/

Grammar Argumented LSTM Neural Networks with Note-Level Encoding for Music Composition

• arxiv: https://arxiv.org/abs/1611.05416

用TensorFlow生成周杰伦歌词

• blog: http://leix.me/2016/11/28/tensorflow-lyrics-generation/
• github: https://github.com/leido/char-rnn-cn

Hip-Hop - Generating lyrics with RNNs

• blog: http://affinelayer.com/hiphop/index.html

Metis Final Project: Music Composition with LSTMs

http://blog.naoya.io/metis-final-project-music-composition-with-lstms/

Neural Translation of Musical Style

• blog: http://imanmalik.com/cs/2017/06/05/neural-style.html
• github: https://github.com/imalikshake/StyleNet

Poetry

NeuralSnap: Generates poetry from images using convolutional and recurrent neural networks

• github: https://github.com/rossgoodwin/neuralsnap

Generating Chinese Classical Poems with RNN Encoder-Decoder

• arxiv: http://arxiv.org/abs/1604.01537

Chinese Poetry Generation with Planning based Neural Network

• intro: COLING 2016. University of Science and Technology of China & Baidu
• arxiv: https://arxiv.org/abs/1610.09889
• blog: http://freecoder.me/archives/213.html

Weiqi (Go)

Teaching Deep Convolutional Neural Networks to Play Go

• arxiv: http://arxiv.org/abs/1412.3409
• demo: https://chrisc36.github.io/deep-go/

Move Evaluation in Go Using Deep Convolutional Neural Networks(Google DeepMind, Google Brain)

• arxiv: http://arxiv.org/abs/1412.6564

Training Deep Convolutional Neural Networks to Play Go

• paper: http://jmlr.org/proceedings/papers/v37/clark15.pdf

Computer Go Research - The Challenges Ahead (Martin Müller. IEEE CIG 2015)

• homepage: https://webdocs.cs.ualberta.ca/~mmueller/talks.html

GoCNN: Using CNN for Go (Weiqi/Baduk) board evaluation with tensorflow

• github: https://github.com/jmgilmer/GoCNN

DarkGo: Go in Darknet

• homepage: http://pjreddie.com/darknet/darkgo-go-in-darknet/

BetaGo: Go bots for the people

• homepage: http://maxpumperla.github.io/betago/
• github: https://github.com/maxpumperla/betago

Deep Learning and the Game of Go

• book: https://www.manning.com/books/deep-learning-and-the-game-of-go
• github: https://github.com//maxpumperla/deep_learning_and_the_game_of_go

DarkForest

Better Computer Go Player with Neural Network and Long-term Prediction (Facebook AI Research)

• arxiv: http://arxiv.org/abs/1511.06410
• github: https://github.com/facebookresearch/darkforestGo
• MIT tech review: http://www.technologyreview.com/view/544181/how-facebooks-ai-researchers-built-a-game-changing-go-engine/

AlphaGo

Mastering the game of Go with deep neural networks and tree search

• intro: AlphaGo. Google DeepMind
• homepage: http://www.deepmind.com/alpha-go.html
• paper: https://storage.googleapis.com/deepmind-data/assets/papers/deepmind-mastering-go.pdf
• naturep page: http://www.nature.com/nature/journal/v529/n7587/full/nature16961.html
• paper: https://gogameguru.com/i/2016/03/deepmind-mastering-go.pdf
• slides: http://www.bioinfo.org.cn/~casp/temp/alphago_slides.pdf
• blog: http://www.furidamu.org/blog/2016/01/26/mastering-the-game-of-go-with-deep-neural-networks-and-tree-search/
• blog(“AlphaGo: From Intuitive Learning to Holistic Knowledge”): https://caminao.wordpress.com/2016/02/01/alphago/
• github: https://github.com/Rochester-NRT/AlphaGo

AlphaGo Teach

• intro: Let the AlphaGo Teaching Tool help you find new and creative ways of playing Go
• homepage: https://alphagoteach.deepmind.com/

AlphaGo的分析

• intro: by 田渊栋
• blog: http://zhuanlan.zhihu.com/yuandong/20607684

How Alphago Works

• slides: http://www.slideshare.net/ShaneSeungwhanMoon/how-alphago-works
• slides: http://pan.baidu.com/s/1qXwagGW

AlphaGo in Depth

• intro: by Mark Chang
• slides: http://www.slideshare.net/ckmarkohchang/alphago-in-depth?qid=283ab3bc-7d04-4e14-a205-b0b671ca4099
• mirror: https://pan.baidu.com/s/1i5JNeRj

Leela

• intro: Leela is a strong Go playing program combining advances in Go programming and further original research into a small, easy to use graphical interface.
• homepage: https://sjeng.org/leela.html

Mastering the game of Go without human knowledge

• nature page: http://www.nature.com/nature/journal/v550/n7676/full/nature24270.html
• paper: https://deepmind.com/documents/119/agz_unformatted_nature.pdf
• notes: https://blog.acolyer.org/2017/11/17/mastering-the-game-of-go-without-human-knowledge/

Computer Go & AlphaGo Zero

• youtube: https://www.youtube.com/watch?v=6fKG4wJ7uBk
• mirror: https://www.bilibili.com/video/av16428694/
• slides: https://drive.google.com/file/d/1rmUyIitEmAtMUKdKEnlHRfmXtpyoxxey/view

AlphaZero: Mastering Games without Human Knowledge - NIPS 2017

• intro: Keynote by David Silver on AlphaGo, AlphaGo Zero and AlphaZero, at the 2017 NIPS Deep Reinforcement Learning Symposium, 6 Dec, Long Beach, CA
• youtube: https://www.youtube.com/watch?v=A3ekFcZ3KNw
• mirror: https://www.bilibili.com/video/av17210816/

PhoenixGo

• intro: Go AI program which implement the AlphaGo Zero paper
• github: https://github.com/Tencent/PhoenixGo

The future is here – AlphaZero learns chess

https://en.chessbase.com/post/the-future-is-here-alphazero-learns-chess

AlphaGo Zero Cheat Sheet

https://applied-data.science/static/main/res/alpha_go_zero_cheat_sheet.png

Chess

Giraffe: Using Deep Reinforcement Learning to Play Chess

• intro: MSc thesis
• arxiv: http://arxiv.org/abs/1509.01549

Spawkfish: neural network based chess engine

• homepage: http://spawk.fish/

Chess position evaluation with convolutional neural network in Julia

• blog: http://int8.io/chess-position-evaluation-with-convolutional-neural-networks-in-julia/

Deep Learning for … Chess

• blog: http://blog.yhat.com/posts/deep-learning-chess.html
• github: https://github.com/erikbern/deep-pink

DeepChess: End-to-End Deep Neural Network for Automatic Learning in Chess

• intro: Winner of Best Paper Award in ICANN 2016
• arxiv: https://arxiv.org/abs/1711.09667
• paper: http://www.cs.tau.ac.il/~wolf/papers/deepchess.pdf
• github: https://github.com/mr-press/DeepChess

Mastering Chess and Shogi by Self-Play with a General Reinforcement Learning Algorithm

• intro: DeepMind
• arxiv: https://arxiv.org/abs/1712.01815

Game

Learning Game of Life with a Convolutional Neural Network

• blog: http://danielrapp.github.io/cnn-gol/
• github: https://github.com/DanielRapp/cnn-gol

Reinforcement Learning using Tensor Flow: A deep Q learning demonstration using Google Tensorflow

• github: https://github.com/nivwusquorum/tensorflow-deepq

Poker-CNN: A Pattern Learning Strategy for Making Draws and Bets in Poker Games Using Convolutional Networks

• arxiv: http://arxiv.org/abs/1509.06731
• paper: http://colinraffel.com/publications/aaai2016poker.pdf
• github: https://github.com/moscow25/deep_draw
• slides: https://drive.google.com/file/d/0B5eOIUHA0khiMjN1YnEtZHMwams/view
• slides: http://pan.baidu.com/s/1nu5zpZ7

TorchCraft: a Library for Machine Learning Research on Real-Time Strategy Games

• intro: Connecting Torch to StarCraft
• arxiv: https://arxiv.org/abs/1611.00625
• github: https://github.com/TorchCraft/TorchCraft

BlizzCon 2016 DeepMind and StarCraft II Deep Learning Panel Transcript

• part 1: http://starcraft.blizzplanet.com/blog/comments/blizzcon-2016-deepmind-and-starcraft-ii-deep-learning-panel-transcript
• part 2: http://starcraft.blizzplanet.com/blog/comments/blizzcon-2016-deepmind-and-starcraft-ii-deep-learning-panel-transcript/2

DeepStack: Expert-Level Artificial Intelligence in No-Limit Poker

• arxiv: https://arxiv.org/abs/1701.01724
• github: https://github.com/lifrordi/DeepStack-Leduc

Gym StarCraft: StarCraft environment for OpenAI Gym, based on Facebook’s TorchCraft

• intro: Gym StarCraft is an environment bundle for OpenAI Gym. It is based on Facebook’s TorchCraft, which is a bridge between Torch and StarCraft for AI research.
• github: https://github.com/deepcraft/gym-starcraft

Multiagent Bidirectionally-Coordinated Nets for Learning to Play StarCraft Combat Games

https://arxiv.org/abs/1703.10069

Learning Macromanagement in StarCraft from Replays using Deep Learning

• intro: CIG 2017. IT University of Copenhagen
• arxiv: https://arxiv.org/abs/1707.03743

Multi-platform Version of StarCraft: Brood War in a Docker Container: Technical Report

• intro: Czech Technical University in Prague
• arxiv: https://arxiv.org/abs/1801.02193
• gihtub: https://github.com/Games-and-Simulations/sc-docker

Macro action selection with deep reinforcement learning in StarCraft

• intro: Bilibili & Nanjing University
• arxiv: https://arxiv.org/abs/1812.00336
• github: https://github.com/Bilibili/LastOrder

DeepLeague

DeepLeague: leveraging computer vision and deep learning on the League of Legends mini map + giving away a dataset of over 100,000 labeled images to further esports analytics research

• blog: https://medium.com/@farzatv/deepleague-leveraging-computer-vision-and-deep-learning-on-the-league-of-legends-mini-map-giving-d275fd17c4e0

DeepLeague (Part 2): The Technical Details

• blog: https://medium.com/@farzatv/deepleague-part-2-the-technical-details-374439e7e09a
• github: https://github.com/farzaa/DeepLeague

Courses

Learning Machines

http://www.patrickhebron.com/learning-machines/

Learning Bit by Bit

https://itp.nyu.edu/varwiki/Syllabus/LearningBitbyBitS10

MACHINE LEARNING FOR MUSICIANS AND ARTISTS (Course opens January 2016)

https://www.kadenze.com/courses/machine-learning-for-musicians-and-artists/info

Machine learning for artists @ ITP-NYU, Spring 2016

• videos/lectures/course notes: http://ml4a.github.io/classes/itp-S16/
• index: http://ml4a.github.io/index/
• github: https://github.com/ml4a/ml4a.github.io
• notes: http://www.kdnuggets.com/2016/04/machine-learning-artists-video-lectures-notes.html
• blog: https://medium.com/@genekogan/machine-learning-for-artists-e93d20fdb097#.25w95beqb

Machine Learning for Artists @ OpenDot, November 2016

• homepage: http://ml4a.github.io/classes/opendot/

The Neural Aesthetic @ SchoolOfMa, Summer 2016

http://ml4a.github.io/classes/neural-aesthetic/

Blogs

Review of machine / deep learning in an artistic context

https://medium.com/@memoakten/machine-deep-learning-in-an-artistic-context-441f28774bcc#.gegpq99ag

Apprentice Work

https://www.technologyreview.com/s/600762/apprentice-work/

Exploring the Intersection of Art and Machine Intelligence

http://googleresearch.blogspot.jp/2016/02/exploring-intersection-of-art-and.html

Using machine learning to generate music

http://www.datasciencecentral.com/profiles/blogs/using-machine-learning-to-generate-music

art in the age of machine intelligence

https://medium.com/artists-and-machine-intelligence/what-is-ami-ccd936394a83#.hyt4ei9a9

Understanding Aesthetics with Deep Learning

https://devblogs.nvidia.com/parallelforall/understanding-aesthetics-deep-learning/

Go, Marvin Minsky, and the Chasm that AI Hasn’t Yet Crossed

blog: https://medium.com/backchannel/has-deepmind-really-passed-go-adc85e256bec#.inx8nfid0

A Return to Machine Learning

• intro: This post is aimed at artists and other creative people who are interested in a survey of recent developments in machine learning research that intersect with art and culture.
• blog: https://medium.com/@kcimc/a-return-to-machine-learning-2de3728558eb#.bp2b1ax2x

Resources

Music, Art and Machine Intelligence Workshop 2016

• youtube: https://www.youtube.com/playlist?list=PL8h7Hk91ScJ2NKFS_i8y4DYk4Zg1LHn0a