Today we will learn how to choose the right Machine Learning metric depending on the problem you are solving and to compute it. A metric gives an idea of how good the model performs. Depending on working on a classification problem or a regression problem the metrics considered are different. It is important to understand that all metrics are just metrics, not the truth.
Today we will learn how to choose the right Machine Learning metric depending on the problem you are solving and to compute it. A metric gives an idea of how good the model performs. Depending on working on a classification problem or a regression problem the metrics considered are different. It is important to understand that all metrics are just metrics, not the truth.
- **F1 score**, **accuracy**, **precision**, **recall** and **AUC scores**. Even if it not considered as a metric, the **confusion matrix** is always useful to understand the model performance.
- **F1 score**, **accuracy**, **precision**, **recall** and **AUC scores**. Even if it not considered as a metric, the **confusion matrix** is always useful to understand the model performance.
Warning: **Imbalanced data set**
Warning: **Imbalanced data set**
Let us assume we are predicting a rare event that occurs less than 2% of the time. Having a model that scores a good accuracy is easy, it doesn't have to be "smart", all it has to do is to always predict the majority class. Depending on the problem it can be disastrous. For example, working with real life data, breast cancer prediction is an imbalanced problem where predicting the majority leads to disastrous consequences. That is why metrics as AUC are useful.
Let us assume we are predicting a rare event that occurs less than 2% of the time. Having a model that scores a good accuracy is easy, it doesn't have to be "smart", all it has to do is to always predict the majority class. Depending on the problem it can be disastrous. For example, working with real life data, breast cancer prediction is an imbalanced problem where predicting the majority leads to disastrous consequences. That is why metrics as AUC are useful.
The goal of this exercise is to learn to use `sklearn.metrics` to compute the mean squared error (MSE).
The goal of this exercise is to learn to use `sklearn.metrics` to compute the mean squared error (MSE).
1. Compute the MSE using `sklearn.metrics` on `y_true` and `y_pred` below:
```
```python
y_true = [91, 51, 2.5, 2, -5]
y_pred = [90, 48, 2, 2, -4]
```
## Correction
1. This question is validated if the MSE outputted is **2.25**.
## Correction
1. This question is validated if the MSE outputted is **2.25**.
# Exercise 2 Accuracy Scikit-learn
The goal of this exercise is to learn to use `sklearn.metrics` to compute the accuracy.
The goal of this exercise is to learn to use `sklearn.metrics` to compute the accuracy.
1. Compute the accuracy using `sklearn.metrics` on `y_true` and `y_pred` below:
```
```python
y_pred = [0, 1, 0, 1, 0, 1, 0]
y_true = [0, 0, 1, 1, 1, 1, 0]
```
## Correction
1. This question is validated if the accuracy outputted is **0.5714285714285714**.
# Exercise 3 Regression
## Correction
The goal of this exercise is to learn to evaluate a machine learning model using many regression metrics.
1. This question is validated if the accuracy outputted is **0.5714285714285714**.
Preliminary:
# Exercise 3 Regression
- Import California Housing data set and split it in a train set and a test set (10%). Fit a linear regression on the data set. *The goal is focus on the metrics, that is why the code to fit the Linear Regression is given.*
The goal of this exercise is to learn to evaluate a machine learning model using many regression metrics.
Preliminary:
- Import California Housing data set and split it in a train set and a test set (10%). Fit a linear regression on the data set. *The goal is focus on the metrics, that is why the code to fit the Linear Regression is given.*
```
#imports
```python
# imports
from sklearn.datasets import fetch_california_housing
from sklearn.model_selection import train_test_split
2. This question is validated if the results match this output:
```
```console
r2 on the train set: 0.3552292936915783
MAE on the train set: 0.5300159371615256
MSE on the train set: 0.5210784446797679
@ -136,24 +128,19 @@ pipe.fit(X_train, y_train)
r2 on the test set: 0.30265471284464673
MAE on the test set: 0.5454023699809112
MSE on the test set: 0.5537420654727396
```
This result shows that the model has slightly better results on the train set than the test set. That's frequent since it is easier to get a better grade on an exam we studied than an exam that is different from what was prepared. However, the results are not good: r2 ~ 0.3. Fitting non linear models as the Random Forest on this data may improve the results. That's the goal of the exercise 5.
This result shows that the model has slightly better results on the train set than the test set. That's frequent since it is easier to get a better grade on an exam we studied than an exam that is different from what was prepared. However, the results are not good: r2 ~ 0.3. Fitting non linear models as the Random Forest on this data may improve the results. That's the goal of the exercise 5.
# Exercise 4 Classification
The goal of this exercise is to learn to evaluate a machine learning model using many classification metrics.
The goal of this exercise is to learn to evaluate a machine learning model using many classification metrics.
Preliminary:
Preliminary:
- Import Breast Cancer data set and split it in a train set and a test set (20%). Fit a logistic regression on the data set. *The goal is focus on the metrics, that is why the code to fit the logistic Regression is given.*
- Import Breast Cancer data set and split it in a train set and a test set (20%). Fit a linear regression on the data set. *The goal is focus on the metrics, that is why the code to fit the Linear Regression is given.*
```
```python
from sklearn.linear_model import LogisticRegression
from sklearn.datasets import load_breast_cancer
from sklearn.model_selection import train_test_split
@ -166,28 +153,27 @@ scaler = StandardScaler()
X_train_scaled = scaler.fit_transform(X_train)
classifier = LogisticRegression()
classifier.fit(X_train_scaled, y_train)
```
1. Predict on the train set and test set
2. Compute F1, accuracy, precision, recall, roc_auc scores on the train set and test set. Print the confusion matrix on the test set results.
2. Compute F1, accuracy, precision, recall, roc_auc scores on the train set and test set. Print the confusion matrix on the test set results.
**Note: AUC can only be computed on probabilities, not on classes.**
3. Plot the AUC curve for on the test set using roc_curve of scikit learn. There many ways to create this plot. It should look like this:
3. Plot the AUC curve for on the test set using roc_curve of scikit learn. There many ways to create this plot. It should look like this:
Having a 99% ROC AUC is not usual. The data set we used is easy to classify. On real data sets, always check if there's any leakage while having such a high ROC AUC score.
Having a 99% ROC AUC is not usual. The data set we used is easy to classify. On real data sets, always check if there's any leakage while having such a high ROC AUC score.
# Exercise 5 Machine Learning models
The goal of this exercise is to have an overview of the existing Machine Learning models and to learn to call them from scikit learn.
We will focus on:
The goal of this exercise is to have an overview of the existing Machine Learning models and to learn to call them from scikit learn.
All these algorithms exist in two versions: regression and classification. Even if the logic is similar in both classification and regression, the loss function is specific to each case.
All these algorithms exist in two versions: regression and classification. Even if the logic is similar in both classification and regression, the loss function is specific to each case.
It is really easy to get lost among all the existing algorithms. This article is very useful to have a clear overview of the models and to understand which algorithm use and when. https://towardsdatascience.com/how-to-choose-the-right-machine-learning-algorithm-for-your-application-1e36c32400b9
Preliminary:
Preliminary:
- Import California Housing data set and split it in a train set and a test set (10%). Fit a linear regression on the data set. *The goal is to focus on the metrics, that is why the code to fit the Linear Regression is given.*
- Import California Housing data set and split it in a train set and a test set (10%). Fit a linear regression on the data set. *The goal is to focus on the metrics, that is why the code to fit the Linear Regression is given.*
```
#imports
```python
# imports
from sklearn.datasets import fetch_california_housing
from sklearn.model_selection import train_test_split
1. Create 5 pipelines with 5 different models as final estimator (keep the imputer and scaler unchanged):
1. Linear Regression
2. SVM
3. Decision Tree (set `random_state=43`)
4. Random Forest (set `random_state=43`)
5. Gradient Boosting (set `random_state=43`)
1. Linear Regression
2. SVM
3. Decision Tree (set `random_state=43`)
4. Random Forest (set `random_state=43`)
5. Gradient Boosting (set `random_state=43`)
Take time to have basic understanding of the role of the basic hyperparameter and their default value.
Take time to have basic understanding of the role of the basic hyperparameters and their defaut value.
- For each algorithm, print the R2, MSE and MAE on both train set and test set.
## Correction
## Correction
1. Some of the algorithms use random steps (random sampling used by the `RandomForest`). I used `random_state = 43` for the Random Forest, the Decision Tree and the Gradient Boosting. This question is validated of the scores you got are close to:
1. Some of the algorithms use random steps (random sampling used by the `RandomForest`). I used `random_state = 43` for the Random Forest, the Decision Tree and the Gradient Boosting. This question is validated of the scores you got are close to:
```
```console
# Linear regression
TRAIN
@ -360,104 +339,93 @@ Take time to have basic understanding of the role of the basic hyperparameters a
MSE on the test set: 0.27058170064218096
```
It is important to notice that the Decision Tree overfits very easily. It learns easily the training data but is not able to extrapolate on the test set. This algorithm is not used a lot.
However, Random Forest and Gradient Boosting propose a solid approach to correct the overfitting (in that case the parameters `max_depth` is set to None that is why the Random Forest overfits the data). These two algorithms are used intensively in Machine Learning Projets.
# Exercise 6 Grid Search
It is important to notice that the Decision Tree over fits very easily. It learns easily the training data but is not able to extrapolate on the test set. This algorithm is not used a lot.
The goal of this exercise is to learn how to make an exhaustive search over specified parameter values for an estimator. This is very useful because the hyperparameters which are the paremeters of the model impact the performance of the model.
The scikit learn object that runs the Grid Search is called GridSearchCV. We will learn tomorrow about the cross validation. For now, let us set the parameter **cv** to `[(np.arange(18576), np.arange(18576,20640))]`.
This means that GridSearchCV splits the data set in a train and test set.
However, Random Forest and Gradient Boosting propose a solid approach to correct the over fitting (in that case the parameters `max_depth` is set to None that is why the Random Forest over fits the data). These two algorithms are used intensively in Machine Learning Projects.
Preliminary:
# Exercise 6 Grid Search
- Load the California Housing data set. As precised, this time, there's no need to split the data set in train set and test set since GridSearchCV does it.
The goal of this exercise is to learn how to make an exhaustive search over specified parameter values for an estimator. This is very useful because the hyperparameter which are the parameters of the model impact the performance of the model.
You will have to run a Grid Search on the Random Forest on at least the hyperparameters that are mentioned below. It doesn't mean these are the only hyperparameters of the model. If possible, try at least 3 different values for each hyperparameters.
The scikit learn object that runs the Grid Search is called GridSearchCV. We will learn tomorrow about the cross validation. For now, let us set the parameter **cv** to `[(np.arange(18576), np.arange(18576,20640))]`.
This means that GridSearchCV splits the data set in a train and test set.
1. Run a Grid Search with `n_jobs` set to `-1` to parallelize the computations on all CPUs. The hyperparameters to change are: n_estimators, max_depth, min_samples_leaf. It may take
Preliminary:
- Load the California Housing data set. As precised, this time, there's no need to split the data set in train set and test set since GridSearchCV does it.
You will have to run a Grid Search on the Random Forest on at least the hyperparameter that are mentioned below. It doesn't mean these are the only hyperparameter of the model. If possible, try at least 3 different values for each hyperparameter.
Now, let us analyse the grid search's results in order to select the best model.
1. Run a Grid Search with `n_jobs` set to `-1` to parallelize the computations on all CPUs. The hyperparameter to change are: n_estimators, max_depth, min_samples_leaf. It may take
2. Write a function that takes as input the Grid Search object and that returns the best model **fitted**, the best set of hyperpameters and the associated score:
Now, let us analyse the grid search's results in order to select the best model.
2. Write a function that takes as input the Grid Search object and that returns the best model **fitted**, the best set of hyperparameter and the associated score:
```
```python
def select_model_verbose(gs):
return trained_model, best_params, best_score
```
3. Use the trained model to predict on a new point:
```
3. Use the trained model to predict on a new point:
How do we know the best model returned by GridSearchCV is good enough and stable ? That is what we will learn tomorrow !
How do we know the best model returned by GridSearchCV is good enough and stable ? That is what we will learn tomorrow !
WARNING:
Some combinations of hyper parameters are not possible. For example using the SVM, the kernel linear has no parameter gamma.
Note:
- GridSearchCV can also take a Pipeline instead of a Machine Learning model. It is useful to combine some Imputers or Dimension reduction techniques with some Machine Learning models in the same Pipeline.
- It may be useful to check on Kaggle if some Kagglers share their Grid Searches.
**WARNING: Some combinations of hyper parameters are not possible. For example using the SVM, the kernel linear has no parameter gamma.**
**Note**:
Ressources:
- GridSearchCV can also take a Pipeline instead of a Machine Learning model. It is useful to combine some Imputers or Dimension reduction techniques with some Machine Learning models in the same Pipeline.
- It may be useful to check on Kaggle if some Kagglers share their Grid Searches.
In my case, the gridsearch parameters are not interesting. Even if I reduced the overfitting of the Random Forest, the score on the test is lower than the score on the test returned by the Gradient Boosting in the previous exercise without optimal parameters search.
2. This question is validated if the function is:
```python
def select_model_verbose(gs):
3. This question is validated if the code used is:
In my case, the `gridsearch` parameters are not interesting. Even if I reduced the over fitting of the Random Forest, the score on the test is lower than the score on the test returned by the Gradient Boosting in the previous exercise without optimal parameters search.
brad-gh
3 years ago
committed by