You've probably seen images liket the ones below that represent machine learning models, but what do they actually mean? The goal of this module is to begin to build some intuition about these models.
Neural networks are a sub-category of machine learning models.
The diagrams below all represent the simplest category of neural networks called a Feed Forward Networks (FFNN). You will also encounter the related names Fully Connected Neural Network (FCNN) and Multilayer Perceptron (MLP).
Nerual Networks come in many shapes and sizes.
In practice, networks can have many millions of nodes.
"Feed Forward" and "Fully Connected"
Can you explain that makes these networks fully connected? What makes them feed forward?
Inputs and Outputs
In machine learning, we need to represent our input and output data with numbers. There are lots of different ways to do that. Any of methods below are valid targets for the inputs and outputs of a neural network.
MIDI Note Number
Frequency (Hz)
Spectrograms
Audio waveforms (Wavenet did this, but there are now better ways to generate waveforms)
Take a look at the output of the output of the matrix above. Data flowing through a neural network is typically represented as a column vector.
z=18170
What we want is a array or probabilities for each possible output. That is, we want a vector that sums to 1. A naive way to do this would be to divide each value by the sum of the vector.
18+17+01818+17+01718+17+00≈0.510.490
Here is how this would be written in machine learning resources:
y^i=∑j=1Kzjzi
In practice, we use a slightly different approach called the softmax function. For an input vector, z=[z1,z2,…,zK], the softmax function is defined as:
y^i=softmax(zi)=∑j=1Kezjezi
This evaluates to a probability distribution, which is a vector of probabilities that sum to 1.
Notice how the softmax exaggarates the difference between the values? This is one of the reasons it is preferred over the naive approach.
Loss
Now that we have a probability distribution, there is one more function we need to complete our model. The loss function measures how "wrong" our model weights are. Classifications models and generative models like the Transformer often use the cross-entropy loss function.
For a single example with a true one-hot label vector y=[y1,…,yK] and predicted probabilities y^=[y^1,…,y^K], the cross-entropy loss is:
L(y,y^)=−i=1∑Kyilog(y^i)
With a one-hot label, only the correct class c has yc=1 (all others are 0), so this reduces to the negative log-likelihood of the model’s probability for the correct class: