The goal of this day is to learn to use Keras to build Neural Networks.
@ -16,14 +15,16 @@ The sequential API allows you to create models layer-by-layer for most problems.
## Historical
## Rules
The correction will provide the code and output because it is not straightforward to reproduce results using Keras. There are many source of randomness. Even if all the seeds are fixed to a constant they may be other source of randomness. https://machinelearningmastery.com/reproducible-results-neural-networks-keras/
The correction will provide the code and output because it is not straightforward to reproduce results using Keras. There are many source of randomness. Even if all the seeds are fixed to a constant they may be other source of randomness.
@ -35,13 +36,11 @@ The goal of this exercise is to learn to call the object `Sequential`.
1. This question is validated if the output is: `<tensorflow.python.keras.engine.sequential.Sequential object at xxx`
# Exercise 2 Dense
The goal of this exercise is to learn to create layers of neurons. Keras proposes options to create custom layers. The neural networks build in these exercises do not require custom layers. `Dense` layers do the job. A dense layer is simply a layer where each unit or neuron is connected to each neuron in the next layer. As seen yesterday, there are three main types of layers: input, hidden and output. The **input layer** that specifies the number of inputs (features) is not represented as a layer in Keras. However, `Dense` has a parameter `input_dim` that gives the number of inputs in the previous layer. The output layer as any hidden layer can be created using `Dense`, the only difference is that the output layer contains one single neuron.
The goal of this exercise is to learn to create layers of neurons. Keras proposes options to create custom layers. The neural networks build in these exercises do not require custom layers. `Dense` layers do the job. A dense layer is simply a layer where each unit or neuron is connected to each neuron in the next layer. As seen yesterday, there are three main types of layers: input, hidden and output. The **input layer** that specifies the number of inputs (features) is not represented as a layer in Keras.
However, `Dense` has a parameter `input_dim` that gives the number of inputs in the previous layer. The output layer as any hidden layer can be created using `Dense`, the only difference is that the output layer contains one single neuron.
1. Create a `Dense` layer with these parameters and return the output of `get_config`:
@ -49,7 +48,6 @@ The goal of this exercise is to learn to create layers of neurons. Keras propose
- 8 neurons
- sigmoid as activation function
2. Create a `Dense` layer with these parameters and return the output of `get_config`:
- Hidden layer (not the first one)
@ -66,7 +64,7 @@ The goal of this exercise is to learn to create layers of neurons. Keras propose
1. This question is validated if the fields`batch_input_shape`,`units` and `activation` match this output:
```
```console
{'name': 'dense_7',
'trainable': True,
'batch_input_shape': (None, 5),
@ -86,7 +84,7 @@ The goal of this exercise is to learn to create layers of neurons. Keras propose
2. This question is validated if the fields`units` and `activation` match this output:
```
```console
{'name': 'dense_8',
'trainable': True,
'dtype': 'float32',
@ -102,9 +100,10 @@ The goal of this exercise is to learn to create layers of neurons. Keras propose
'kernel_constraint': None,
'bias_constraint': None}
```
3. This question is validated if the fields`units` and `activation` match this output:
```
```console
{'name': 'dense_9',
'trainable': True,
'dtype': 'float32',
@ -136,34 +135,39 @@ The goal of this exercise is to combine the layers and to create a neural networ
1. This question is validated if the code that creates the neural network is:
The first two layers could use another activation function that sigmoid (eg: relu)
# Exercise 4 Optimize
The goal of this exercise is to learn to train the neural network. Once the architecture of the neural network is set there are two steps to train the neural network:
- `compile`: The compilation step aims to set the loss function, to choose the algoithm to minimize the chosen loss function and to choose the metric the model outputs.
- `compile`: The compilation step aims to set the loss function, to choose the algorithm to minimize the chosen loss function and to choose the metric the model outputs.
- The **optimizer**. We’ll stick with a pretty good default: the Adam gradient-based optimizer. Keras has many other optimizers you can look into as well.
- The **loss function**. Depending on the problem to solve: classification or regression Keras proposes different loss functions. In classification Keras distinguishes between `binary_crossentropy` (2 classes) and `categorical_crossentropy` (>2 classes), so we’ll use the latter.
- The **metric(s)**. A list of metrics. Depending on the problem to solve: classification or regression Keras proposes different loss functions. For example for classification the metric can be the accuracy.
- The **loss function**. Depending on the problem to solve: classification or regression Keras proposes different loss functions. In classification Keras distinguishes between `binary_crossentropy` (2 classes) and `categorical_crossentropy` (>2 classes), so we will use the latter.
- The **metric(s)**. A list of metrics. Depending on the problem to solve: classification or regression Keras proposes different loss functions. For example for classification the metric can be the accuracy.
- `fit`: Training a model in Keras literally consists only of calling fit() and specifying some parameters. There are a lot of possible parameters, but we’ll only manually supply a few:
- The **training data**, commonly known as X and Y, respectively.
- The **number of epochs** (iterations over the entire dataset) to train for.
- The **batch size** (number of samples per gradient update) to use when training.
This article gives more details about **epoch** and **batch size**: https://machinelearningmastery.com/difference-between-a-batch-and-an-epoch/
This article gives more details about **epoch** and **batch size**:
1. Create the following neural network (classification):
- Set the right number of inputs variables
- hidden layer 1: 10 neurons and sigmoid as activation function.
- hidden layer 2: 5 neurons and sigmoid as activation function.
@ -178,7 +182,7 @@ The goal of this exercise is to learn to train the neural network. Once the arch
1. This question is validated if the output of `model.get_config()['layers']` matches the fields `batch_input_shape`, `units` and `activation`.
```
```console
[{'class_name': 'InputLayer',
'config': {'batch_input_shape': (None, 30),
'dtype': 'float32',
@ -232,12 +236,12 @@ The goal of this exercise is to learn to train the neural network. Once the arch
'kernel_constraint': None,
'bias_constraint': None}}]
```
You should notice that the neural network is struggling to learn. By luck the initialization of the weights might have led to an accuracy close of 90%. But when I trained the neural network, with `batch_size=300` on the data here is the ouptput of the last epoch (50):
You should notice that the neural network is struggling to learn. By luck the initialization of the weights might have led to an accuracy close of 90%. But when I trained the neural network, with `batch_size=300` on the data here is the output of the last epoch (50):
lee
3 years ago