deep learning

References

Summary

Hyperparameters: values set before training

Parameters: values that are set by the model during training

GPUs are really useful for neural networks, because they're designed to be really good at calculations with matrices (which show up in graphics)

where v is the input vector (technically you also have to factor in the activation functions, so at each new step you really have f(vi), where f is the activation function applied element-wise and vi is the output of the previous layer)

Autoencoder

Fewer neurons in the hidden layer than the input layer, and is trained to output the input

Decoder

Language generation is a kind of decoder

Similarly, translation encodes in one language and decodes in the other

Clustering

A kind of unsupervised learning

Support Vector Machine

Neural Instruction, Multiple Data

Each processor corresponds to a single neuron

History

Threshold Logic Units (1940s – 1960s)

Most models in this period used boolean inputs and outputs and threshold activation functions

They had a single layer

The focus was on if neural networks could learn basic logical relations like AND and OR

1943

A Logical Calculus of the Ideas Immanent in Nervous Activity Walter McCulloch and Walter Pitts

1949

The Organization of Behaviour, D. O. Hebb

1958

Rosenblatt's Perceptron Model

Perceptron Training Rule

Perceptron Convergence Theorem

1960

Widrow and Hoff

The point of this model was that you could use it to train the network to produce any output not just  ± 1

1969

Perceptrons, Minsky and Papert

To fix this, people began using multiple layers of neurons, but they didn't know how to train them

Connectionism (1980s – 1990s)

1980

Fukushima

1982

Hopfield

1986

Parallel Distributed Processing Research Group

The problem is that this method was incredibly inefficient for deeper networks

1989

LeCun

CNNs are designed for feature detection

1990

Elman

RNNs

1991

Hochreiter

1997

Hochreiter and Schmidhuber

LSTMs

Deep Learning (2000s – 2010s)

Greedy layer-wise pretraining

Glorot initialization

2011

Rectified Linear Activation Function

2014

Sequence-to-sequence