Deep Learning and Autonomous Driving

Courses

(Toronto) CSC2541: Visual Perception for Autonomous Driving, Winter 2016

• homepage: http://www.cs.toronto.edu/~urtasun/courses/CSC2541/CSC2541_Winter16.html

(MIT) 6.S094: Deep Learning for Self-Driving Cars

• homepage: http://selfdrivingcars.mit.edu/
• github: https://github.com/lexfridman/deepcars
• youtube: https://www.youtube.com/playlist?list=PLrAXtmErZgOeiKm4sgNOknGvNjby9efdf
• mirror: https://pan.baidu.com/s/1boLRFaB

How to Land An Autonomous Vehicle Job: Coursework

• blog: https://medium.com/self-driving-cars/how-to-land-an-autonomous-vehicle-job-coursework-e7acc2bfe740#.7vfjx3i1j

Papers

An Empirical Evaluation of Deep Learning on Highway Driving

• arxiv: http://arxiv.org/abs/1504.01716
• github: https://github.com/brodyh/caffe

Real-time Joint Object Detection and Semantic Segmentation Network for Automated Driving

• intro: NeurIPS 2018 Workshop on Machine Learning on the Phone and other Consumer Devices (MLPCD 2)
• arxiv: https://arxiv.org/abs/1901.03912

Optical Flow augmented Semantic Segmentation networks for Automated Driving

• intro: VISAPP 2019 Oral
• arxiv: https://arxiv.org/abs/1901.07355

AuxNet: Auxiliary tasks enhanced Semantic Segmentation for Automated Driving

• intro: Short Paper for a poster presentation at VISAPP 2019
• arxiv: https://arxiv.org/abs/1901.05808

Design of Real-time Semantic Segmentation Decoder for Automated Driving

• intro: VISAPP 2019
• arxiv: https://arxiv.org/abs/1901.06580

Hierarchical Multi-task Deep Neural Network Architecture for End-to-End Driving

https://arxiv.org/abs/1902.03466

DeepDriving

DeepDriving: Learning Affordance for Direct Perception in Autonomous Driving

• project page: http://deepdriving.cs.princeton.edu/
• paper: http://deepdriving.cs.princeton.edu/paper.pdf
• code: http://deepdriving.cs.princeton.edu/DeepDriving.zip

End to End Learning for Self-Driving Cars

• intro: NVIDIA DevBox and Torch 7, 30 FPS
• arxiv: http://arxiv.org/abs/1604.07316
• blog: https://devblogs.nvidia.com/parallelforall/deep-learning-self-driving-cars/
• demo: https://www.youtube.com/watch?v=NJU9ULQUwng&feature=youtu.be
• github: https://github.com/SullyChen/Nvidia-Autopilot-TensorFlow

End-to-End Deep Learning for Self-Driving Cars

• blog: https://devblogs.nvidia.com/parallelforall/deep-learning-self-driving-cars/

---

Can we unify monocular detectors for autonomous driving by using the pixel-wise semantic segmentation of CNNs?

• arxiv: http://arxiv.org/abs/1607.00971

BRAIN4CARS: Cabin Sensing for Safe and Personalized Driving

Brain4Cars: Sensory-Fusion Recurrent Neural Models for Driver Activity Anticipation

Brain4Cars: Car That Knows Before You Do via Sensory-Fusion Deep Learning Architecture

• arxiv: http://arxiv.org/abs/1601.00740

Car that Knows Before You Do: Anticipating Maneuvers via Learning Temporal Driving Models

• arxiv: http://arxiv.org/abs/1504.02789
• github: https://github.com/asheshjain399/ICCV2015_Brain4Cars

Recurrent Neural Networks for Driver Activity Anticipation via Sensory-Fusion Architecture

• project page: http://www.brain4cars.com/
• arxiv: http://arxiv.org/abs/1509.05016
• github: https://github.com/asheshjain399/RNNexp

Long-term Planning by Short-term Prediction

• arxiv: http://arxiv.org/abs/1602.01580

Learning a Driving Simulator

• introo: by hacker Geohot
• project page: http://research.comma.ai/
• arxiv: http://arxiv.org/abs/1608.01230
• paper: https://github.com/commaai/research/blob/master/paper/commalds.pdf
• github: https://github.com/commaai/research

Comma.ai open-sources the data it used for its first successful driverless trips

• blog: https://techcrunch.com/2016/08/03/comma-ai-open-sources-the-data-it-used-for-its-first-successful-driverless-trips/

Autonomous driving challenge: To Infer the property of a dynamic object based on its motion pattern using recurrent neural network

• arxiv: http://arxiv.org/abs/1609.00361

Safe, Multi-Agent, Reinforcement Learning for Autonomous Driving

• arxiv: https://arxiv.org/abs/1610.03295

Learning from Maps: Visual Common Sense for Autonomous Driving

• arxiv: https://arxiv.org/abs/1611.08583

SAD-GAN: Synthetic Autonomous Driving using Generative Adversarial Networks

• intro: Accepted at the Deep Learning for Action and Interaction Workshop, 30th Conference on Neural Information Processing Systems (NIPS 2016)
• arxiv: https://arxiv.org/abs/1611.08788

MultiNet: Real-time Joint Semantic Reasoning for Autonomous Driving

• intro: first place on Kitti Road Segmentation. joint classification, detection and semantic segmentation via a unified architecture, less than 100 ms to perform all tasks
• arxiv: https://arxiv.org/abs/1612.07695
• github: https://github.com/MarvinTeichmann/MultiNet

Interpretable Learning for Self-Driving Cars by Visualizing Causal Attention

• intro: UC Berkeley
• arxiv: https://arxiv.org/abs/1703.10631

Virtual to Real Reinforcement Learning for Autonomous Driving

• intro: Shanghai Jiao Tong University & UC Berkeley & Tsinghua University
• arxiv: https://arxiv.org/abs/1704.03952

Computer Vision for Autonomous Vehicles: Problems, Datasets and State-of-the-Art

• homepage: http://www.cvlibs.net/projects/autonomous_vision_survey/
• arxiv: https://arxiv.org/abs/1704.05519

Deep Reinforcement Learning framework for Autonomous Driving

https://arxiv.org/abs/1704.02532

Systematic Testing of Convolutional Neural Networks for Autonomous Driving

https://arxiv.org/abs/1708.03309

MODNet: Moving Object Detection Network with Motion and Appearance for Autonomous Driving

https://arxiv.org/abs/1709.04821

CFENet: An Accurate and Efficient Single-Shot Object Detector for Autonomous Driving

• intro: CVPR 2018 Workshop of Autonomous Driving (WAD)
• arxiv: https://arxiv.org/abs/1806.09790

LaneNet: Real-Time Lane Detection Networks for Autonomous Driving

• intro: Duke University & Horizon Robotics, Inc.
• arxiv: https://arxiv.org/abs/1807.01726

Learning End-to-end Autonomous Driving using Guided Auxiliary Supervision

https://arxiv.org/abs/1808.10393

Rethinking Self-driving: Multi-task Knowledge for Better Generalization and Accident Explanation Ability

• intro: Waseda University
• arxiv: https://arxiv.org/abs/1809.11100
• demo: https://www.youtube.com/watch?v=N7ePnnZZwdE

Pixel and Feature Level Based Domain Adaption for Object Detection in Autonomous Driving

https://arxiv.org/abs/1810.00345

Multi-task Learning with Attention for End-to-end Autonomous Driving

• intro: CVPR 2021 Workshop on Autonomous Driving
• arxiv: https://arxiv.org/abs/2104.10753

MP3: A Unified Model to Map, Perceive, Predict and Plan

• intro: Uber ATG & University of Toronto
• arxiv: https://arxiv.org/abs/2101.06806

Level 2 Autonomous Driving on a Single Device: Diving into the Devils of Openpilot

• intro: Shanghai AI Laboratory & Shanghai Jiao Tong University & UCSD & SenseTime
• arxiv: https://arxiv.org/abs/2206.08176
• github: https://github.com/OpenPerceptionX/Openpilot-Deepdive

Real-time Full-stack Traffic Scene Perception for Autonomous Driving with Roadside Cameras

• intro: ICRA 2022
• intro: University of Michigan & Ford Motor Company
• arxiv: https://arxiv.org/abs/2206.09770

ST-P3: End-to-end Vision-based Autonomous Driving via Spatial-Temporal Feature Learning

• intro: ECCV 2022
• intro: Shanghai Jiao Tong University & Shanghai AI Laboratory & The University of California & JD Explore Academy
• arxiv: https://arxiv.org/abs/2207.07601
• github: https://github.com/OpenPerceptionX/ST-P3

Effective Adaptation in Multi-Task Co-Training for Unified Autonomous Driving

• intro: NeurIPS 2022
• intro: Shenzhen Campus of Sun Yat-Sen University & Huawei Noah’s Ark Lab
• arxiv: https://arxiv.org/abs/2209.08953

Planning-oriented Autonomous Driving

• intro: CVPR 2023 best paper
• intro: Shanghai AI Laboratory, Wuhan University, SenseTime Research
• project page: https://opendrivelab.github.io/UniAD/
• arxiv: https://arxiv.org/abs/2212.10156
• github: https://github.com/OpenDriveLab/UniAD

Projects

Caffe-Autopilot: Car autopilot software that uses C++, BVLC Caffe, OpenCV, and SFML

• github: https://github.com/SullyChen/Caffe-Autopilot

Self Driving Car Demo

• intro; A project that trains a virtual car to how to move an object around a screen (drive itself) without running into obstacles using a type of reinforcement learning called Q-Learning
• github: https://github.com/llSourcell/Self-Driving-Car-Demo/

Autoware: Open-source software for urban autonomous driving

• github: https://github.com/CPFL/Autoware

Open Sourcing 223GB of Driving Data

• homepage: https://udacity.com/self-driving-car
• blog: https://medium.com/udacity/open-sourcing-223gb-of-mountain-view-driving-data-f6b5593fbfa5#.q8nk5bfpp
• github: https://github.com/udacity/self-driving-car

Machine Learning for RC Cars

• github: https://github.com/kendricktan/suiron

Self Driving (Toy) Ferrari

• github: https://github.com/RyanZotti/Self-Driving-Car

Lane Finding Project for Self-Driving Car ND

• github: https://github.com/udacity/CarND-LaneLines-P1

Instructions on how to get your development environment ready for Udacity Self Driving Car (SDC) Challenges

• github: https://github.com/gtarobotics/self-driving-car

DeepDrive: self-driving car AI

• intro: Caffe Model / Dataset / Tips and Tricks
• homepage: http://deepdrive.io/

DeepDrive setup: Run a self-driving car simulator from the comfort of your own PC

• github: https://github.com/crizCraig/deepdrive

DeepTesla: End-to-End Learning from Human and Autopilot Driving

http://selfdrivingcars.mit.edu/deeptesla/

DeepPicar: A Low-cost Deep Neural Network-based Autonomous Car

• arxiv: https://arxiv.org/abs/1712.08644
• github: https://github.com//heechul/picar

Autonomous Driving in Reality with Reinforcement Learning and Image Translation

• intro: Shanghai Jiao Tong University
• arxiv: https://arxiv.org/abs/1801.05299

End-to-end Multi-Modal Multi-Task Vehicle Control for Self-Driving Cars with Visual Perception

https://arxiv.org/abs/1801.06734

Blogs

Self-driving cars: How far away are we REALLY from autonomous cars?(7 Aug 2015)

http://www.alphr.com/cars/1001329/self-driving-cars-how-far-away-are-we-really-from-autonomous-cars

Practice makes perfect: Driverless cars will learn from their mistakes(9 Oct 2015)

http://www.alphr.com/cars/1001713/practice-makes-perfect-driverless-cars-will-learn-from-their-mistakes

Eyes on the Road: How Autonomous Cars Understand What They’re Seeing

• blog: http://blogs.nvidia.com/blog/2016/01/05/eyes-on-the-road-how-autonomous-cars-understand-what-theyre-seeing/

Human-in-the-loop deep learning will help drive autonomous cars

http://venturebeat.com/2016/06/25/human-in-the-loop-deep-learning-will-help-drive-autonomous-cars/

Using reinforcement learning in Python to teach a virtual car to avoid obstacles

• part 1: https://medium.com/@harvitronix/using-reinforcement-learning-in-python-to-teach-a-virtual-car-to-avoid-obstacles-6e782cc7d4c6#.rneyuerga
• part 2: https://medium.com/@harvitronix/reinforcement-learning-in-python-to-teach-a-virtual-car-to-avoid-obstacles-part-2-93e614fcd238#.1pt1lli4c
• part 3: https://medium.com/@harvitronix/reinforcement-learning-in-python-to-teach-an-rc-car-to-avoid-obstacles-part-3-a1d063ac962f#.jwzm2v1r4
• github: https://github.com/harvitronix/reinforcement-learning-car

Autonomous RC car using Raspberry Pi and Neural Networks

• blog: http://www.multunus.com/blog/2016/07/autonomous-rc-car-using-raspberry-pi-and-neural-networks/
• github: https://github.com/multunus/autonomous-rc-car

The Road Ahead: Autonomous Vehicles Startup Ecosystem

https://medium.com/the-mission/the-road-ahead-autonomous-vehicles-startup-ecosystem-3c91d546673d#.gft1xyh9l

Deep Driving - A revolutionary AI technique is about to transform the self-driving car

https://www.technologyreview.com/s/602600/deep-driving/

Visualizations for regressing wheel steering angles in self driving cars with Keras

• blog: http://jacobcv.blogspot.jp/2016/10/visualizations-for-regressing-wheel.html
• github: https://github.com/jacobgil/keras-steering-angle-visualizations