Last week we saw some ways to evaluate a machine learning model but they were only for classification problems. Let's see another way to evaluate one and then focus on regression model evaluations.
Table of contents:
- Confusion matrix
- Regression model evaluators
- Using the scoring parameter
The confusion matrix is a way to compare what the machine predicted with what it was supposed to predict, it gets this name because lets us know where the model is getting confused and predict a label instead of another. Using the same estimator of the last week we can already import what we need:
from sklearn.model_selection import train_test_split from sklearn.svm import SVC X = iris_df.drop('target', axis = 1) y = iris_df['target'] X_train, X_test, y_train, y_test = train_test_split(X, y, test_size = .15) clf = SVC() clf.fit(X_train, y_train) y_preds = clf.predict(X_test)
now we can import the function to do the confusion matrix:
But what does this mean? It is based on the false positive rates (fpr) and true positive rates (tpr), which we saw in the last post. Every cell of the matrix indicates if the model predicted correctly or not the class. We can have a quick overview with the pandas function crosstab:
Knowing that 0, 1, and 2 are our labels Setosa, Versicolor, and Virginica, we can see where the model predicted a right label and where a wrong one.
Having saved the result of the
confusion_matrix function into
conf_mat we can even import the seaborn library to plot it in an utterly fashion way:
Till now we saw just how to evaluate a classification model and how to use a binary classification evaluator for multi-class classification. Evaluating a regression model is much simpler than what we already did despite being vital to machine learning in general
the coefficient of determination or R^2 (R-squared) is the
score method that we saw from the begin of the serie, but let's understand how does it work. Let's begin importing a regression dataset:
you can find the data here
looking at the data we know that we have all the cases of chickenpox in Hungary divided by county areas. We want to predict the total cases based on how many cases were in every county. We'll create column with the total cases for every day, to do so we need a single line of code:
chickenpox['total'] = chickenpox.sum(axis = 1)
The result is the same which we already saw thousands of times: a number between 0.0 and 1.0 which correspond to a percentage. The coefficient of determination compares every prediction with the test label. We can use the dedicated function from the library to calculate it:
the mean absolute error score calculates the absolute difference between every actual value and every predicted one:
Absolute means that every number is positive, so a complete way to do the dataframe would be:
To complete this evaluation we have to just calculate the mean of the
difference column. Let's compare our calcs with the dedicated function:
And from what we can see we did everything right.
The MAE tells us the range of the error of our model, through this evaluation, we now know that the errors of our model range between 74 and -74.
- R-squared is used for accuracy but doesn't tell you where the model is doing wrong.
- MAE gives a good indication of how wrong your model is
- MSE amplifies larger differences than MAE
In this case, if 200 cases are twice as bad as 100 cases we could use the MAE, but if a disease has a very high rate of transmission then 200 cases would be more than twice as bad as 100 and we should pay more attention to the MSE.
for the full list, you can go here
We saw all the most important ways to evaluate our machine learning model avoiding using mindlessly the metrics functions without knowing what they do. If you have any doubt feel free to leave a comment.