Numbers/weights/parameters are discrete, but the representation is distributed
If each number is the distance along an axis, then the set of numbers forms a “space” – a so called “latent space” that encodes “deeper information” – meaning?
Research in this field is all about finding “good numbers” to perform well doing “things we care about”
“Neural Networks”
a.k.a:
Feed-Forward Neural Networks (FF)
Fully-Connected Neural Networks (FCN)
Artificial Neural Networks (ANN)
Multi-Layered Perceptrons (MLP)
Learning
Learning!
Errors – Landscape
Loss functions
Error surfaces
Optimization challenges
Loss
Loss Function – a.k.a. “cost function”, “error”: “Distance Measures” – our “gap”
A cost function looks like a smooth, curved line, across a graph. For any point on this line, the height above zero represents “cost” in some way – e.g. how much “energy” the model will incur when tuned to a specific setting. The higher the point, the farther from ideal (of 0) the system is. We want to find the lowest point on this line, where the cost is smallest.
Common loss functions include:
Mean Square Error for regression tasks: \(MSE = \frac{1}{n} \sum_{i-1}^{n}(Y_i - \hat{Y}_{i})^{2}\)
Gradient descent algorithms “feel their way” to the bottom of the curve by picking a point and calculating the slope (or gradient) of the curve around it, and then moving in the direction where the slope is steepest
Imagine this as feeling your way down a mountain in the dark. You may not know exactly where to move, or how close to valley floor you will get, but if, in general, you head down the slope in the steepest direction, you would hope to arrive at the lowest point in the area
.. the “outside function” is \(f()\) and the “inside function” is \(g()\)
Chain rule translates as: “the derivate of the ‘outside’” \(\times\) “the derivative of the ‘inside’”
Alternatively,
If \(y = f[g(x)]\)
Then \(y' = f'[g(x)].g'(x)\)
Training Lifecycle
Forward pass
Backward pass (backpropagation)
Weight updates – until convergence
Backpropagation
Training hidden layers – “breaking free from Minski’s XOR cage”
Backpropagation is a gradient estimation method used to train neural network models. The gradient estimate is used by the optimization algorithm to compute the network parameter updates.
It is an efficient application of the chain rule to such networks (known since 1673). It is also known as the reverse mode of automatic differentiation.
Automatic differentiation is a powerful software technique used to learn (any) parameter from error gradients
The phrase “‘’Deep Learning’’” really applies to any design of neural network that makes use of many layers – today, numbering 10s, 100s or even 1000s of layers.
The explosion of interest in Deep Learning began with the success of AlexNet in a machine vision competition (circa. 2012)
Our “latent space”, which encodes our ”deeper information”, abstracts – meaning as we penetrate deeper into the network*
Automatic feature engineering
In the early history of neural computation, the computational power obtained by stacking multiple layers was not fully recognised, and in any case, few researchers had access to the scale of compute necessary to explore these advantages. Early neural networks used around 3-5 layers.
Transformers
A Transformer block is an Attention layer + MLP layer
A Transformer stacks multiple Transformer blocks”
Attention
Code
Basic neural network implementation
Key components and functions
import numpy as np# f = x * x -- i.e. no bias termX = np.array([1, 2, 3, 4], dtype=np.float32)y = np.array([2, 4, 6, 8], dtype=np.float32)w =0.0
# model predictiondef forward(x):return w*x
# loss = MSE -- in the case of linear regressiondef loss(y, y_predicted):return ((y_predicted - y)**2).mean()
