Neural Networks

You've probably seen images liket the ones below that represent machine learning models, but what do they actually mean? The goal of this module is to begin to build some intuition about these models.

Neural networks are a sub-category of machine learning models.

The diagrams below all represent the simplest category of neural networks called a Feed Forward Networks (FFNN). You will also encounter the related names Fully Connected Neural Network (FCNN) and Multilayer Perceptron (MLP).

Nerual Networks come in many shapes and sizes.

In practice, networks can have many millions of nodes.

"Feed Forward" and "Fully Connected"

Can you explain that makes these networks fully connected? What makes them feed forward?

Inputs and Outputs

In machine learning, we need to represent our input and output data with numbers. There are lots of different ways to do that. Any of methods below are valid targets for the inputs and outputs of a neural network.

  • MIDI Note Number
  • Frequency (Hz)
  • Spectrograms
  • Audio waveforms (Wavenet did this, but there are now better ways to generate waveforms)
  • ABC notation (via tokens)
  • What are some others?
Some formats work better than others!24-166614-8-642145

Think of each blue dot as a function that takes multiple inputs and produces 1 output value.

Think of the entire network as a function that makes multiple inputs and produces multiple outputs.

Weights

How did we calculate the 24 in the first hidden layer?

Where do these orange values come from?

24-1666141.0-3.01.02.00.02.0-2.00.01.01.03.0-1.01.00.02.01.01.0-2.02.05.0-1.01.00.00.0-1.01.01.00.0-1.02.0-8-642145

⚠️ Some steps where omitted! ⚠️
The functions in the hidden layer do a little more than this... but we will get to that later.

A more practical example

Let's discuss how we could create a neural network that generates chord sequences.

-3.644.25-4.163.95-0.84-4.23-2.531.94-0.10.6-0.5-0.1-0.0-0.60.8-0.30.0-0.4-0.3-0.0-0.50.70.30.30.50.20.3-0.4-0.80.3-0.6-0.40.5-0.50.70.8-0.6-0.40.1-0.20.70.5-0.30.4-0.50.0-0.40.1-0.20.00.50.60.20.70.20.00.50.70.0-0.6-0.6-0.6-0.6-0.7-1.460.63-1.64-1.04-2.06-0.51-2.781451

Points so far

  • A neural network is a function that takes multiple inputs and produces multiple outputs.
  • There are two kinds of values that we need to think about: data and parameters
    • Data is the information flowing through the network (inputs, activations, and outputs).
    • Parameters are the internal values of the network (weights and biases) that are updated during training.

Neurons with bias

Remember I said that we skipped over some of the details of the neurons? Let us add it for completeness.

+1.0+0.0+3.0-1.0+0.0-4.0+1.0+3.021-1676111.0-3.01.02.00.02.0-2.00.01.01.03.0-1.01.00.02.01.01.0-2.02.05.0-1.01.00.00.0-1.01.01.00.0-1.02.0-14-635153

Neurons with bias and "activation"

The activation function is the final missing piece of our neuron model.
Activation:
+1.0+0.0+3.0-1.0+0.0-4.0+1.0+3.021076111.0-3.01.02.00.02.0-2.00.01.01.03.0-1.01.00.02.01.01.0-2.02.05.0-1.01.00.00.0-1.01.01.00.0-1.02.018170153

Softmax

Take a look at the output of the output of the matrix above. Data flowing through a neural network is typically represented as a column vector.
z=[18170]z = {\def\arraystretch{1}\begin{bmatrix} 18 \\ 17 \\ 0 \end{bmatrix}}
What we want is a array or probabilities for each possible output. That is, we want a vector that sums to 11. A naive way to do this would be to divide each value by the sum of the vector.
[1818+17+01718+17+0018+17+0]≈[0.510.490]{\def\arraystretch{1.8}\begin{bmatrix} \frac{18}{18 + 17 + 0} \\ \frac{17}{18 + 17 + 0} \\ \frac{0}{18 + 17 + 0} \end{bmatrix}} \approx {\def\arraystretch{1.8}\begin{bmatrix} 0.51 \\ 0.49 \\ 0 \end{bmatrix}}
Here is how this would be written in machine learning resources:
y^i=zi∑j=1Kzj\hat{y}_i = \frac{z_i}{\sum_{j=1}^{K} z_j}
In practice, we use a slightly different approach called the softmax function.
For an input vector, z=[z1,z2,…,zK]\mathbf{z}=[z_1,z_2,…,z_K], the softmax function is defined as:
y^i=softmax(zi)=ezi∑j=1Kezj\hat{y}_i = \text{softmax}(z_i) = \frac{e^{z_i}}{\sum_{j=1}^{K} e^{z_j}}
This evaluates to a probability distribution, which is a vector of probabilities that sum to 1.
y^=[e18e18+e17+e0e17e18+e17+e0e0e18+e17+e0]≈[0.730.270.00]\hat{y} = {\def\arraystretch{2}\begin{bmatrix} \frac{e^{18}}{e^{18} + e^{17} + e^{0}} \\ \frac{e^{17}}{e^{18} + e^{17} + e^{0}} \\ \frac{e^{0}}{e^{18} + e^{17} + e^{0}} \end{bmatrix}} \approx {\def\arraystretch{2}\begin{bmatrix} 0.73 \\ 0.27 \\ 0.00 \end{bmatrix}}
Notice how the softmax exaggarates the difference between the values? This is one of the reasons it is preferred over the naive approach.

Loss

Now that we have a probability distribution, there is one more function we need to complete our model. The loss function measures how "wrong" our model weights are. Classifications models and generative models like the Transformer often use the cross-entropy loss function.

For a single example with a true one-hot label vector y=[y1,…,yK]y = [y_1, \dots, y_K] and predicted probabilities y^=[y^1,…,y^K]\hat{y} = [\hat{y}_1, \dots, \hat{y}_K], the cross-entropy loss is:

L(y,y^)=−∑i=1Kyilog⁡(y^i)L(y, \hat{y}) = - \sum_{i=1}^{K} y_i \log(\hat{y}_i)

With a one-hot label, only the correct class cc has yc=1y_c = 1 (all others are 0), so this reduces to the negative log-likelihood of the model’s probability for the correct class:

y=[100]y = {\def\arraystretch{1}\begin{bmatrix} 1 \\ 0 \\ 0 \end{bmatrix}}
L(y,y^)=−log⁡(y^c)L(y, \hat{y}) = - \log(\hat{y}_c)
Example: L=−log⁡(0.73)≈0.313\text{Example: } L = -\log(0.73) \approx 0.313

Resources