The goal of this day is to learn to use Keras to build Neural Networks.
The goal of this day is to learn to use Keras to build Neural Networks.
There are two ways to build Keras models: sequential and functional.
There are two ways to build Keras models: sequential and functional.
The sequential API allows you to create models layer-by-layer for most problems. It is limited in that it does not allow you to create models that share layers or have multiple inputs or outputs. The exercises focuses on the usage of the sequential API.
The sequential API allows you to create models layer-by-layer for most problems. It is limited in that it does not allow you to create models that share layers or have multiple inputs or outputs. The exercises focuses on the usage of the sequential API.
'2.4.3'
'2.4.3'
## Historical
## Historical
## Rules
## Rules
The correction will provide the code and output because it is not straightforward to reproduce results using Keras. There are many source of randomness. Even if all the seeds are fixed to a constant they may be other source of randomness. https://machinelearningmastery.com/reproducible-results-neural-networks-keras/
The correction will provide the code and output because it is not straightforward to reproduce results using Keras. There are many source of randomness. Even if all the seeds are fixed to a constant they may be other source of randomness.
The goal of this exercise is to learn to call the object `Sequential`.
The goal of this exercise is to learn to call the object `Sequential`.
1. Put the object Sequential in a variable named `model` and print the variable `model`.
1. Put the object Sequential in a variable named `model` and print the variable `model`.
## Correction
## Correction
1. This question is validated if the output is: `<tensorflow.python.keras.engine.sequential.Sequential object at xxx`
1. This question is validated if the output is: `<tensorflow.python.keras.engine.sequential.Sequential object at xxx`
# Exercise 2 Dense
# Exercise 2 Dense
The goal of this exercise is to learn to create layers of neurons. Keras proposes options to create custom layers. The neural networks build in these exercises do not require custom layers. `Dense` layers do the job. A dense layer is simply a layer where each unit or neuron is connected to each neuron in the next layer. As seen yesterday, there are three main types of layers: input, hidden and output. The **input layer** that specifies the number of inputs (features) is not represented as a layer in Keras. However, `Dense` has a parameter `input_dim` that gives the number of inputs in the previous layer. The output layer as any hidden layer can be created using `Dense`, the only difference is that the output layer contains one single neuron.
The goal of this exercise is to learn to create layers of neurons. Keras proposes options to create custom layers. The neural networks build in these exercises do not require custom layers. `Dense` layers do the job. A dense layer is simply a layer where each unit or neuron is connected to each neuron in the next layer. As seen yesterday, there are three main types of layers: input, hidden and output. The **input layer** that specifies the number of inputs (features) is not represented as a layer in Keras.
1. Create a `Dense` layer with these parameters and return the output of `get_config`:
However, `Dense` has a parameter `input_dim` that gives the number of inputs in the previous layer. The output layer as any hidden layer can be created using `Dense`, the only difference is that the output layer contains one single neuron.
- First hidden layer connected to 5 input variables.
1. Create a `Dense` layer with these parameters and return the output of `get_config`:
- 8 neurons
- sigmoid as activation function
- First hidden layer connected to 5 input variables.
- 8 neurons
- sigmoid as activation function
2. Create a `Dense` layer with these parameters and return the output of `get_config`:
2. Create a `Dense` layer with these parameters and return the output of `get_config`:
- Hidden layer (not the first one)
- Hidden layer (not the first one)
- 4 neurons
- 4 neurons
- sigmoid as activation function
- sigmoid as activation function
3. Create a `Dense` layer with these parameters and return the output of `get_config`:
3. Create a `Dense` layer with these parameters and return the output of `get_config`:
- Output layer
- Output layer
- 1 neuron
- 1 neuron
- sigmoid as activation function
- sigmoid as activation function
## Correction
## Correction
1. This question is validated if the fields`batch_input_shape`,`units` and `activation` match this output:
1. This question is validated if the fields`batch_input_shape`,`units` and `activation` match this output:
The first two layers could use another activation function that sigmoid (eg: relu)
The first two layers could use another activation function that sigmoid (eg: relu)
# Exercise 4 Optimize
# Exercise 4 Optimize
The goal of this exercise is to learn to train the neural network. Once the architecture of the neural network is set there are two steps to train the neural network:
The goal of this exercise is to learn to train the neural network. Once the architecture of the neural network is set there are two steps to train the neural network:
- `compile`: The compilation step aims to set the loss function, to choose the algorithm to minimize the chosen loss function and to choose the metric the model outputs.
- `compile`: The compilation step aims to set the loss function, to choose the algoithm to minimize the chosen loss function and to choose the metric the model outputs.
- The **optimizer**. We’ll stick with a pretty good default: the Adam gradient-based optimizer. Keras has many other optimizers you can look into as well.
- The **optimizer**. We’ll stick with a pretty good default: the Adam gradient-based optimizer. Keras has many other optimizers you can look into as well.
- The **loss function**. Depending on the problem to solve: classification or regression Keras proposes different loss functions. In classification Keras distinguishes between `binary_crossentropy` (2 classes) and `categorical_crossentropy` (>2 classes), so we will use the latter.
- The **loss function**. Depending on the problem to solve: classification or regression Keras proposes different loss functions. In classification Keras distinguishes between `binary_crossentropy` (2 classes) and `categorical_crossentropy` (>2 classes), so we’ll use the latter.
- The **metric(s)**. A list of metrics. Depending on the problem to solve: classification or regression Keras proposes different loss functions. For example for classification the metric can be the accuracy.
- The **metric(s)**. A list of metrics. Depending on the problem to solve: classification or regression Keras proposes different loss functions. For example for classification the metric can be the accuracy.
- `fit`: Training a model in Keras literally consists only of calling fit() and specifying some parameters. There are a lot of possible parameters, but we’ll only manually supply a few:
- `fit`: Training a model in Keras literally consists only of calling fit() and specifying some parameters. There are a lot of possible parameters, but we’ll only manually supply a few:
- The **training data**, commonly known as X and Y, respectively.
- The **training data**, commonly known as X and Y, respectively.
- The **number of epochs** (iterations over the entire dataset) to train for.
- The **number of epochs** (iterations over the entire dataset) to train for.
- The **batch size** (number of samples per gradient update) to use when training.
- The **batch size** (number of samples per gradient update) to use when training.
This article gives more details about **epoch** and **batch size**: https://machinelearningmastery.com/difference-between-a-batch-and-an-epoch/
This article gives more details about **epoch** and **batch size**:
1. Create the following neural network (classification):
1. Create the following neural network (classification):
- Set the right number of inputs variables
- hidden layer 1: 10 neurons and sigmoid as activation function.
- hidden layer 2: 5 neurons and sigmoid as activation function.
- output layer: 1 neuron and sigmoid as activation function.
- Choose the accuracy metric, the adam optimizer, the adapted loss and epoch smaller than 50.
Import the breast cancer data set from `sklearn.datasets` using `load_breast_cancer` and train the neural network on the data set.
- Set the right number of inputs variables
- hidden layer 1: 10 neurons and sigmoid as activation function.
- hidden layer 2: 5 neurons and sigmoid as activation function.
- output layer: 1 neuron and sigmoid as activation function.
- Choose the accuracy metric, the adam optimizer, the adapted loss and epoch smaller than 50.
2. Scale the data using `StandardScaler` from `sklearn.preprocessing`. Train the neural network again.
Import the breast cancer data set from `sklearn.datasets` using `load_breast_cancer` and train the neural network on the data set.
2. Scale the data using `StandardScaler` from `sklearn.preprocessing`. Train the neural network again.
## Correction
## Correction
1. This question is validated if the output of `model.get_config()['layers']` matches the fields `batch_input_shape`, `units` and `activation`.
1. This question is validated if the output of `model.get_config()['layers']` matches the fields `batch_input_shape`, `units` and `activation`.
```
```console
[{'class_name': 'InputLayer',
[{'class_name': 'InputLayer',
'config': {'batch_input_shape': (None, 30),
'config': {'batch_input_shape': (None, 30),
'dtype': 'float32',
'dtype': 'float32',
@ -232,9 +239,10 @@ The goal of this exercise is to learn to train the neural network. Once the arch
'kernel_constraint': None,
'kernel_constraint': None,
'bias_constraint': None}}]
'bias_constraint': None}}]
```
```
You should notice that the neural network is struggling to learn. By luck the initialization of the weights might have led to an accuracy close of 90%. But when I trained the neural network, with `batch_size=300` on the data here is the ouptput of the last epoch (50):
You should notice that the neural network is struggling to learn. By luck the initialization of the weights might have led to an accuracy close of 90%. But when I trained the neural network, with `batch_size=300` on the data here is the output of the last epoch (50):
The goal of this day is to learn to use Keras to build Neural Networks and train them on small data sets.
classification & regression
The goal of this day is to learn to use Keras to build Neural Networks and train them on small data sets.
classification & regression
'2.4.3'
'2.4.3'
## Historical
## Historical
## Rules
## Rules
The correction will provide the code and output because it is not straightforward to reproduce results using Keras. There are many source of randomness. Even if all the seeds are fixed to a constant they may be other source of randomness. https://machinelearningmastery.com/reproducible-results-neural-networks-keras/
The correction will provide the code and output because it is not straightforward to reproduce results using Keras. There are many source of randomness. Even if all the seeds are fixed to a constant they may be other source of randomness. https://machinelearningmastery.com/reproducible-results-neural-networks-keras/
The goal of this exercise is to learn to set up the optimization for a regression neural network. There's no code to run in that exercise. In W2D2E3, we implemented a neural network designed for regression. We will be using this neural network:
The goal of this exercise is to learn to set up the optimization for a regression neural network. There's no code to run in that exercise. In W2D2E3, we implemented a neural network designed for regression. We will be using this neural network:
As a reminder, the main difference between a regression and classification neural network's architecture is the output layer activation function.
As a reminder, the main difference between a regression and classification neural network's architecture is the output layer activation function.
1. Fill this chunk of code to set up the optimization part of the regression neural network:
1. Fill this chunk of code to set up the optimization part of the regression neural network:
```
```python
model.compile(
model.compile(
optimizer='adam',
optimizer='adam',
loss='',#TODO1
loss='',#TODO1
metrics=[''] #TODO2
metrics=[''] #TODO2
)
)
```
```
Hint:
- Mean Squared Error (MSE) and Mean Absolute Error (MAE) are common loss functions used for regression problems. Mean Absolute Error is less sensitive to outliers. Different loss functions are used for classification problems. Similarly, evaluation metrics used for regression differ from classification.
Hint:
- Mean Squared Error (MSE) and Mean Absolute Error (MAE) are common loss functions used for regression problems. Mean Absolute Error is less sensitive to outliers. Different loss functions are used for classification problems. Similarly, evaluation metrics used for regression differ from classification.
https://keras.io/api/metrics/regression_metrics/
https://keras.io/api/metrics/regression_metrics/
## Correction:
## Correction
1. This question is validated if the chunk of code is:
1. This question is validated if the chunk of code is:
```
```python
model.compile(
model.compile(
optimizer='adam',
optimizer='adam',
loss='mse',
loss='mse',
metrics=['mse']
metrics=['mse']
)
)
```
```
All regression metrics or losses used are correct. As explained before, the loss functions are chosen thanks to nice mathematical properties. That is why most of the time the loss function used for regression is the MSE or MAE.
https://keras.io/api/losses/regression_losses/
All regression metrics or losses used are correct. As explained before, the loss functions are chosen thanks to nice mathematical properties. That is why most of the time the loss function used for regression is the MSE or MAE.
The goal of this exercise is to learn to train a neural network to perform a regression on a data set.
The goal of this exercise is to learn to train a neural network to perform a regression on a data set.
The data set is Auto MPG Dataset and the go is to build a model to predict the fuel efficiency of late-1970s and early 1980s automobiles. To do this, provide the model with a description of many automobiles from that time period. This description includes attributes like: cylinders, displacement, horsepower, and weight.
The data set is Auto MPG Dataset and the goal is to build a model to predict the fuel efficiency of late-1970s and early 1980s automobiles. To do this, provide the model with a description of many automobiles from that time period. This description includes attributes like: cylinders, displacement, horsepower, and weight.
- Drop the columns: **model year**, **origin**, **car name**
- Split train test without shuffling the data. Keep 20% for the test set.
- Scale the data using Standard Scaler
- Drop the columns: **model year**, **origin**, **car name**
- Split train test without shuffling the data. Keep 20% for the test set.
- Scale the data using Standard Scaler
2. Train a neural network on the train set and predict on the test set. The neural network should have 2 hidden layers and the loss should be **mean_squared_error**. The expected **mean absolute error** on the test set is maximum 10.
2. Train a neural network on the train set and predict on the test set. The neural network should have 2 hidden layers and the loss should be **mean_squared_error**. The expected **mean absolute error** on the test set is maximum 10.
*Hint*: inscrease the number of epochs
**Warning**: Do no forget to evaluate the neural network on the **SCALED** test set.
*Hint*: increase the number of epochs
**Warning**: Do no forget to evaluate the neural network on the **SCALED** test set.
## Correction
## Correction
1. This question is validated if the input DataFrames are:
1. This question is validated if the input DataFrames are:
X_train_scaled shape is (313, 5) and the first 5 rows are:
X_train_scaled shape is (313, 5) and the first 5 rows are:
@ -102,7 +105,7 @@ X_train_scaled shape is (313, 5) and the first 5 rows are:
The output neuron has to be `Dense(1)` - by defaut the activation funtion is linear. The loss has to be **mean_squared_error** and the **input_dim** has to be **5**. All variations on the others parameters are accepted.
The output neuron has to be `Dense(1)` - by default the activation function is linear. The loss has to be **mean_squared_error** and the **input_dim** has to be **5**. All variations on the others parameters are accepted.
*Hint*: To get the score on the test set, `evaluate` could have been used: `model.evaluate(X_test_scaled, y_test)`.
*Hint*: To get the score on the test set, `evaluate` could have been used: `model.evaluate(X_test_scaled, y_test)`.
# Exercise 3 Multi classification - Softmax
# Exercise 3 Multi classification - Softmax
The goal of this exercise is to learn to a neural network architecture for multi-class data. This is an important type of problem on which to practice with neural networks because the three class values require specialized handling. A multi-classification neural network uses as output layer a **softmax** layer. The **softmax** activation function is an extension of the sigmoid as it is designed to output the probabilities to belong to each class in a multi-class problem. This output layer has to contain as much neurons as classes in the multi-classification problem. This article explains in detail how it works. https://developers.google.com/machine-learning/crash-course/multi-class-neural-networks/softmax
The goal of this exercise is to learn to a neural network architecture for multi-class data. This is an important type of problem on which to practice with neural networks because the three class values require specialized handling. A multi-classification neural network uses as output layer a **softmax** layer. The **softmax** activation function is an extension of the sigmoid as it is designed to output the probabilities to belong to each class in a multi-class problem. This output layer has to contain as much neurons as classes in the multi-classification problem. This article explains in detail how it works.
Let us assume we want to classify images and we know they contain either apples, bears, candies, eggs or dogs (extension of the example in the link above).
1. Create the architecture for a multi-class neural network with the following architecture and return `print(model.summary())`:
Let us assume we want to classify images and we know they contain either apples, bears, candies, eggs or dogs (extension of the example in the link above).
- 5 inputs variables
1. Create the architecture for a multi-class neural network with the following architecture and return `print(model.summary())`:
- hidden layer 1: 16 neurons and sigmoid as activation function
- hidden layer 2: 8 neurons and sigmoid as activation function
- output layer: The number of neurons and the activation function should be adapted to this multi-classification problem.
- 5 inputs variables
- hidden layer 1: 16 neurons and sigmoid as activation function
- hidden layer 2: 8 neurons and sigmoid as activation function
- output layer: The number of neurons and the activation function should be adapted to this multi-classification problem.
## Correction
## Correction
1. This question is validated if the code that creates the neural network is:
1. This question is validated if the code that creates the neural network is:
The goal of this exercise is to learn to optimize a multi-classification neural network. As learnt previously, the loss function used in binary classification is the log loss - also called in Keras `binary_crossentropy`. This function is defined for binary classification and can be extended to multi-classfication. In Keras, the extended loss that supports multi-classification is `binary_crossentropy`. There's no code to run in that exercise.
The goal of this exercise is to learn to optimize a multi-classification neural network. As learnt previously, the loss function used in binary classification is the log loss - also called in Keras `binary_crossentropy`. This function is defined for binary classification and can be extended to multi-classfication. In Keras, the extended loss that supports multi-classification is `binary_crossentropy`. There's no code to run in that exercise.
1. Fill the chunk of code below in order to optimize the neural network defined in the previous exercise. Choose the adapted loss, adam as optimizer and the accuracy as metric.
1. Fill the chunk of code below in order to optimize the neural network defined in the previous exercise. Choose the adapted loss, adam as optimizer and the accuracy as metric.
```
```python
model.compile(loss='',#TODO1
model.compile(loss='',#TODO1
optimizer='', #TODO2
optimizer='', #TODO2
metrics=['']) #TODO3
metrics=['']) #TODO3
```
```
## Correction
## Correction
1. This question is validated if the chunk of code is:
1. This question is validated if the chunk of code is:
The goal of this exercise is to learn to use a neural network to classify a multiclass data set. The data set used is the Iris data set which allows to classify flower given basic features as flower's measurement.
The goal of this exercise is to learn to use a neural network to classify a multiclass data set. The data set used is the Iris data set which allows to classify flower given basic features as flower's measurement.
Preliminary:
Preliminary:
- Split train test. Keep 20% for the test set. Use `random_state=1`.
- Scale the data using Standard Scaler
- Split train test. Keep 20% for the test set. Use `random_state=1`.
- Scale the data using Standard Scaler
1. Use the `LabelBinarizer` from Sckit-learn to create a one hot encoding of the target. As you know, the output layer of a multi-classification neural network shape is equal to the number of classes. The output layer expects to have a target with the same shape as its output layer.
1. Use the `LabelBinarizer` from Sckit-learn to create a one hot encoding of the target. As you know, the output layer of a multi-classification neural network shape is equal to the number of classes. The output layer expects to have a target with the same shape as its output layer.
2. Train a neural network on the train set and predict on the test set. The neural network should have 1 hidden layers. The expected **accuracy** on the test set is minimum 90%.
2. Train a neural network on the train set and predict on the test set. The neural network should have 1 hidden layers. The expected **accuracy** on the test set is minimum 90%.
*Hint*: inscrease the number of epochs
**Warning**: Do no forget to evaluate the neural network on the **SCALED** test set.
*Hint*: increase the number of epochs
**Warning**: Do no forget to evaluate the neural network on the **SCALED** test set.
## Correction
## Correction
1. This question is validated if the output of the first ten values of the train labels are:
1. This question is validated if the output of the first ten values of the train labels are:
```
```console
array([[0, 1, 0],
array([[0, 1, 0],
[0, 0, 1],
[0, 0, 1],
[0, 1, 0],
[0, 1, 0],
@ -231,9 +235,9 @@ array([[0, 1, 0],
2. This question is validated if the accuracy on the test set is bigger than 90%. To evaluate the accuracy on the test set you can use: `model.evaluate(X_test_sc, y_test_multi_class)`.
2. This question is validated if the accuracy on the test set is bigger than 90%. To evaluate the accuracy on the test set you can use: `model.evaluate(X_test_sc, y_test_multi_class)`.
Here is an implementation that gives 96% accuracy on the test set.
Here is an implementation that gives 96% accuracy on the test set.
Deep learning is a huge domain. We will focus on Artificial Neural Networks. The goal is to understand how do the neural networks train and train them on data. Understand the challenges of training a neural network
Deep learning is a huge domain. We will focus on Artificial Neural Networks. The goal is to understand how do the neural networks train and train them on data. Understand the challenges of training a neural network
Architectures as RNN, LSTM (learn sequences, used in TS and NLP),CNN used a lot in image processing are well know algorithms in deep learning but won't be covered by the AI branch. Once you have a good understanding of ANN feel free to extend your knowledge to new architectures.
Architectures as RNN, LSTM (learn sequences, used in TS and NLP), CNN used a lot in image processing are well know algorithms in deep learning but won't be covered by the AI branch. Once you have a good understanding of ANN feel free to extend your knowledge to new architectures.
The goal of this exercise is to understand the role of a neuron and to implement a neuron.
The goal of this exercise is to understand the role of a neuron and to implement a neuron.
An artificial neuron, the basic unit of the neural network, (also referred to as a perceptron) is a mathematical function. It takes one or more inputs that are multiplied by values called “weights” and added together. This value is then passed to a non-linear function, known as an activation function, to become the neuron’s output.
An artificial neuron, the basic unit of the neural network, (also referred to as a perceptron) is a mathematical function. It takes one or more inputs that are multiplied by values called “weights” and added together. This value is then passed to a non-linear function, known as an activation function, to become the neuron’s output.
As desbribed in the article, **a neuron takes inputs, does some math with them, and produces one output**.
As described in the article, **a neuron takes inputs, does some math with them, and produces one output**.
Let us assume there are 2 inputs. Here are the three steps involved in the neuron:
Let us assume there are 2 inputs. Here are the three steps involved in the neuron:
1. Each input is multiplied by a weight
1. Each input is multiplied by a weight:
- x1 -> x1 * w1
- x2 -> x2 * w2
- `x1 -> x1 * w1`
2. The weighted inputs are added together with a biais b
- `x2 -> x2 * w2`
- (x1 * w1) + (x2 * w2) + b
3. The sum is passed through an activation function
2. The weighted inputs are added together with a biais b:
- y = f((x1 * w1) + (x2 * w2) + b)
- The activation function is a function you know from W2DAY2 (Logistic Regression): **the sigmoid**
- `(x1 * w1) + (x2 * w2) + b`
Example:
3. The sum is passed through an activation function:
- `y = f((x1 * w1) + (x2 * w2) + b)`
- The activation function is a function you know from W2DAY2 (Logistic Regression): **the sigmoid**
x1 = 2 , x2 = 3 , w1 = 0, w2= 1, b = 4
Example:
`x1 = 2`, `x2 = 3`, `w1 = 0`, `w2 = 1`, `b = 4`
1. Step 1: Multiply by a weight
1. Step 1: Multiply by a weight
- x1 -> 2 * 0 = 0
- x2 -> 3 * 1 = 3
- `x1 -> 2 * 0 = 0`
- `x2 -> 3 * 1 = 3`
2. Step 2: Add weigthed inputs and bias
2. Step 2: Add weigthed inputs and bias
- 0 + 3 + 4 = 7
- `0 + 3 + 4 = 7`
3. Step 3: Activation function
3. Step 3: Activation function
- y = f(7) = 0.999
- `y = f(7) = 0.999`
---
---
1. Implement a the function feedforward of the class `Neuron` that takes as input the inputs (x1, x2) and that uses the attributes: the weights and the biais to return y:
1. Implement a the function `feedforward` of the class `Neuron` that takes as input the inputs (x1, x2) and that uses the attributes: the weights and the biais to return y:
The goal of this exercise is to understand how to combine three neurons to form a neural network. A neural newtwork is nothing else than neurons connected together. As shown in the figure the neural network is composed of **layers**:
The goal of this exercise is to understand how to combine three neurons to form a neural network. A neural network is nothing else than neurons connected together. As shown in the figure the neural network is composed of **layers**:
- Input layer: it only represents input data. **It doesn't contain neurons**.
- Input layer: it only represents input data. **It doesn't contain neurons**.
- Output layer: it represents the last layer. It contains a neuron (in some cases more than 1).
- Output layer: it represents the last layer. It contains a neuron (in some cases more than 1).
- Hidden layer: any layer between the input (first) layer and output (last) layer. Many hidden layers can be stacked. When there are many hidden layers, the neural networks is deep.
- Hidden layer: any layer between the input (first) layer and output (last) layer. Many hidden layers can be stacked. When there are many hidden layers, the neural networks is deep.
Notice that the neuron **o1** in the output layer takes as input the output of the neurons **h1** and **h2** in the hidden layer.
Notice that the neuron **o1** in the output layer takes as input the output of the neurons **h1** and **h2** in the hidden layer.
1. Implement the function `feedforward` of the class `OurNeuralNetwork` that takes as input the input data and returns the output y. Return the output for these neurons:
1. Implement the function `feedforward` of the class `OurNeuralNetwork` that takes as input the input data and returns the output y. Return the output for these neurons:
# The inputs for o1 are the outputs from h1 and h2
# TODO
return y
```
## Correction
## Correction
1. This question is validated the output is: **0.9524917424084265**
1. This question is validated if for `x1=2` and `x2=3`, the output is: **0.9524917424084265**
# Exercise 3 Log loss
# Exercise 3 Log loss
The goal of this exercise is to implement the Log loss function. As mentioned last week, this function is used in classification as a **loss function**. It means that the better the classifier is, the smaller the loss function is. W2D1, you implemented the gradient descent on the MSE loss to update the weights of the linear regression. Similarly, the minimization of the Log loss leads to finding optimal weights.
The goal of this exercise is to implement the Log loss function. As mentioned last week, this function is used in classification as a **loss function**. It means that the better the classifier is, the smaller the loss function is. W2D1, you implemented the gradient descent on the MSE loss to update the weights of the linear regression. Similarly, the minimization of the Log loss leads to finding optimal weights.
The goal if the network is to predict the success at the exam given math and chemistry grades. The inputs are `math` and `chemistry` and the target is `exam_sucess`.
The goal if the network is to predict the success at the exam given math and chemistry grades. The inputs are `math` and `chemistry` and the target is `exam_sucess`.
1. Compute and return the output of the neural network for each of the students. Here are the weights and biases of the neural network:
1. Compute and return the output of the neural network for each of the students. Here are the weights and biases of the neural network:
```python
neuron_h1 = Neuron(0.05, 0.001, 0)
neuron_h2 = Neuron(0.02, 0.003, 0)
neuron_o1 = Neuron(2,0,0)
```
```
2. Compute the `log_loss` for the data given the output of the neural network with the 4 students.
neuron_h1 = Neuron(0.05, 0.001, 0)
neuron_h2 = Neuron(0.02, 0.003, 0)
neuron_o1 = Neuron(2,0,0)
```
2. Compute the logloss for the data given the output of the neural network with the 4 students.
## Correction
## Correction
1. This question is validated if the output is:
1. This question is validated if the output is:
```
Bob: 0.7855253278357536
Eli: 0.7771516558846259
Tom: 0.8067873659804015
Ryan: 0.7892343955586032
```
2. This question is validated if the logloss for the 4 students is **0.5485133607757963**.
```console
Bob: 0.7855253278357536
Eli: 0.7771516558846259
Tom: 0.8067873659804015
Ryan: 0.7892343955586032
```
# Exercise 5 Regression
2. This question is validated if the `log_loss` for the 4 students is **0.5485133607757963**.
The goal of this exercise is to learn to adapt the output layer to regression.
# Exercise 5 Regression
As a reminder, one of reasons for which the sigmoid is used in classification is because it contracts the output between 0 and 1 which is the expected output range for a probability (W2D2: Logistic regression). However, the output of the regression is not a probability.
In order to perform a regression using a neural network, the activation function of the neuron on the output layer has to be modified to **identity function**. In mathematics, the identity function is: **f(x) = x**. In other words it means that it returns the input as so. The three steps become:
The goal of this exercise is to learn to adapt the output layer to regression.
As a reminder, one of reasons for which the sigmoid is used in classification is because it contracts the output between 0 and 1 which is the expected output range for a probability (W2D2: Logistic regression). However, the output of the regression is not a probability.
In order to perform a regression using a neural network, the activation function of the neuron on the output layer has to be modified to **identity function**. In mathematics, the identity function is: **`f(x) = x`**. In other words it means that it returns the input as so. The three steps become:
1. Each input is multiplied by a weight
1. Each input is multiplied by a weight
- x1 -> x1 * w1
- x2 -> x2 * w2
2. The weighted inputs are added together with a biais b
- (x1 * w1) + (x2 * w2) + b
3. The sum is passed through an activation function
- y = f((x1 * w1) + (x2 * w2) + b)
- The activation function is **the identity**
- y = (x1 * w1) + (x2 * w2) + b
All other neurons' activation function **doesn't change**.
- `x1 -> x1 * w1`
- `x2 -> x2 * w2`
1. Adapt the neuron class implemented in exercise 1. It now takes as a parameter `regression` which is boolean. When its value is `True`, `feedforward` should use the identity function as activation function instead of the sigmoid function.
2. The weighted inputs are added together with a biais `b`
- `(x1 * w1) + (x2 * w2) + b`
```
3. The sum is passed through an activation function
All other neuron's activation function **doesn't change**.
#TODO
return y
```
1. Adapt the neuron class implemented in exercise 1. It now takes as a parameter `regression` which is boolean. When its value is `True`, `feedforward` should use the identity function as activation function instead of the sigmoid function.
2. Now, the goal of the network is to predict the physics' grade at the exam given math and chemistry grades. The inputs are `math` and `chemistry` and the target is `physics`.
```python
neuron = Neuron(0,1,4, True)
neuron.feedforward(2,3)
```
| name | math | chemistry | physics |
2. Now, the goal of the network is to predict the physic's grade at the exam given math and chemistry grades. The inputs are `math` and `chemistry` and the target is `physics`.
The goal of this **1 week** project is to get the highest possible score on a Data Science competition. More precisely you will have to predict who survived the Titanic crash.
The goal of this **1 week** project is to get the highest possible score on a Data Science competition. More precisely you will have to predict who survived the Titanic crash.
@ -8,40 +8,37 @@ The goal of this **1 week** project is to get the highest possible score on a Da
[titanic]: titanic.jpg "Titanic"
[titanic]: titanic.jpg "Titanic"
### Kaggle
### Kaggle
Kaggle is an online community of data scientists and machine learning practitioners. Kaggle allows users to find and publish data sets, explore and build models in a web-based data-science environment, work with other data scientists and machine learning engineers, and enter competitions to solve data science challenges. It’s a crowd-sourced platform to attract, nurture, train and challenge data scientists from all around the world to solve data science, machine learning and predictive analytics problems.
Kaggle is an online community of data scientists and machine learning practitioners. Kaggle allows users to find and publish data sets, explore and build models in a web-based data-science environment, work with other data scientists and machine learning engineers, and enter competitions to solve data science challenges. It’s a crowd-sourced platform to attract, nurture, train and challenge data scientists from all around the world to solve data science, machine learning and predictive analytics problems.
### Titanic - Machine Learning from Disaster
### Titanic - Machine Learning from Disaster
One of the first Kaggle competition I did was: Titanic - Machine Learning from Disaster. This is a not-to-be-missed Kaggle competition.
One of the first Kaggle competition I did was: Titanic - Machine Learning from Disaster. This is a not-to-be-missed Kaggle competition.
https://www.kaggle.com/c/titanic
You can see more [here](https://www.kaggle.com/c/titanic)
The sinking of the Titanic is one of the most infamous shipwrecks in history. On April 15, 1912, during her maiden voyage, the widely considered “unsinkable” RMS Titanic sank after colliding with an iceberg. Unfortunately, there weren’t enough lifeboats for everyone onboard, resulting in the death of 1502 out of 2224 passengers and crew.
The sinking of the Titanic is one of the most infamous shipwrecks in history. On April 15, 1912, during her maiden voyage, the widely considered “unsinkable” RMS Titanic sank after colliding with an iceberg. Unfortunately, there were not enough lifeboats for everyone onboard, resulting in the death of 1502 out of 2224 passengers and crew.
While there was some element of luck involved in surviving, it seems some groups of people were more likely to survive than others.
While there was some element of luck involved in surviving, it seems some groups of people were more likely to survive than others.
In this challenge, you have to build a predictive model that answers the question: **“what sorts of people were more likely to survive?”** using passenger data (ie name, age, gender, socio-economic class, etc).
## Preliminary
In this challenge, you have to build a predictive model that answers the question: **“what sorts of people were more likely to survive?”** using passenger data (ie name, age, gender, socio-economic class, etc). **You will have to submit your prediction on Kaggle**.
The way the Kaggle platform works is explained in the challenge overview page. If you need more details, I suggest this ressource that gives detailed explanations.
The way the Kaggle platform works is explained in the challenge overview page. If you need more details, I suggest this [resource](https://towardsdatascience.com/getting-started-with-kaggle-f9138b35ae18) that gives detailed explanations.
- Create a username following that structure: username_01EDU_ location_MM_YYYY. Submit the description profil and push it on GitHub the first day of the week. Do not touch this file anymore.
- Create a username following this structure: username_01EDU_ location_MM_YYYY. Submit the description profile and push it on GitHub the first day of the week. Do not touch this file anymore.
- It is possible to have different personal accounts merged in a team for one single competition.
- It is possible to have different personal accounts merged in a team for one single competition.
## Deliverables
## Deliverables
```
```console
project
project
│ README.md
│ README.md
│ environment.yml
│ environment.yml
│ username.txt
│ username.txt
│
│
└───data
└───data
@ -55,57 +52,65 @@ project
|───scripts
|───scripts
│
│
```
```
- `README.md` introducts the project, shows the username, describes the features engineering and the best score on the **leaderboard**.
- `README.md` introduction to the project, shows the username, describes the features engineering and the best score on the **leaderboard**.
- `environment.yml` contains all libraries required to run the code.
- `environment.yml` contains all libraries required to run the code.
- `username.txt` contains the username, the last modified date of the file **has to correspond to the first day of the project**.
- `EDA.ipynb` contains the exploratory data analysis. This file is should contain all steps of data analysis that contributed or not to improve the accuracy. It has to be commented so that the reviewer can understand the analysis and run it without any problem.
- `scripts` contains python file(s) that perform(s) the feature engineering, the model's training and prediction on the test set. It could also be one single Jupyter Notebook. It has to be commented to help the reviewers understand the approach and run the code without any bugs.
- `username.txt` contains the username, the last modified date of the file **has to correspond to the first day of the project**.
- `EDA.ipynb` contains the exploratory data analysis. This file is should contain all steps of data analysis that contributed or not to improve the accuracy. It has to be commented so that the reviewer can understand the analysis and run it without any problem.
- `scripts` contains python file(s) that perform(s) the feature engineering, the model's training and prediction on the test set. It could also be one single Jupyter Notebook. It has to be commented to help the reviewers understand the approach and run the code without any bugs.
- **Submit your predictions on the Kaggle's competition platform**. Check your ranking and score in the leaderboard.
### Scores
### Scores
In order to validate the project you will have to score at least **79% accuracy on the Leaderboard**:
In order to validate the project you will have to score at least **79% accuracy on the Leaderboard**:
- 79% accuracy is the minimum score to validate the project.
- 79% accuracy is the minimum score to validate the project.
Scores indication:
Scores indication:
- 79% difficult - minimum required
- 79% difficult - minimum required
- 81% very difficult: smart feature engineering needed
- 81% very difficult: smart feature engineering needed
- More than 83%: excellent that corresponds to the top 2% on Kaggle
- More than 83%: excellent that corresponds to the top 2% on Kaggle
- More than 85%: cheating
- More than 85%: cheating
#### Cheating
#### Cheating
It is impossible to get 100%. Who would have predicted that Rose wouldn't let Jack on the door ? https://www.insider.com/jack-and-rose-werent-on-a-door-in-titanic-2019-7
It is impossible to get 100%. Who would have predicted that Rose wouldn't let [Jack on the door](https://www.insider.com/jack-and-rose-werent-on-a-door-in-titanic-2019-7) ?
All people having 100% of accuracy on the Leaderboard cheated, there's no point to compare with them or to cheat. The Kaggle community estimates that having more than 85% is almost considered as cheated submissions as they are element of luck involved in the surviving.
All people having 100% of accuracy on the Leaderboard cheated, there's no point to compare with them or to cheat. The Kaggle community estimates that having more than 85% is almost considered as cheated submissions as they are element of luck involved in the surviving.
**You can't used external data sets than the ones provided in that competition.**
**You can't used external data sets than the ones provided in that competition.**
## The key points
## The key points
- **Feature engineering**:
- **Feature engineering**:
Imagine yourself as an investigator trying to understand what happened exactly in that boat. Do not hesitate to watch the movie to try to find as many insights as possible. Without the feature engineering there's no way to validate the project ;-)
Imagine yourself as an investigator trying to understand what happened exactly in that boat. Do not hesitate to watch the movie to try to find as many insights as possible. Without the feature engineering there's no way to validate the project ;-)
- The leaderboard evaluates on test data for which you don't have the labels. It means that there's no point to overfit the train set. Check the overfitting on the train set by dividing the data and by cross-validating the accuracy.
- The leaderboard evaluates on test data for which you don't have the labels. It means that there's no point to over fit the train set. Check the over fitting on the train set by dividing the data and by cross-validating the accuracy.
## Advice
## Advice
Don't try to build the perfect model the first day. Iterate a lot and test your assumptions:
Don't try to build the perfect model the first day. Iterate a lot and test your assumptions:
Iteration 1:
Iteration 1:
- Predict all passengers die
- Predict all passengers die
Iteration 2
Iteration 2
- Fit a logisitc regression with a basic feature engineering
Iteration 3:
- Fit a logistic regression with a basic feature engineering
- Perform an EDA. Make assumptions and check them. Example: What if first class passengers survived more. Check the assumption through EDA and create relevant features to help the model capture the information.
- Run a gridsearch
Iteration 3:
- Perform an EDA. Make assumptions and check them. Example: What if first class passengers survived more. Check the assumption through EDA and create relevant features to help the model capture the information.
Badr Ghazlane
3 years ago