5.5. Model evaluation¶

The sklearn.metrics module implements useful functions for assessing the performance of an estimator under a specific criterion. Functions whose name ends with _score return a scalar value to maximize (the higher the better). Functions whose name ends with _error or _loss return a scalar value to minimize (the lower the better).

Note

Estimators usually define a score method which provides a suitable evaluation score for this estimator.

For pairwise metrics, see the Pairwise metrics, Affinities and Kernels section.

5.5.1. Classification metrics¶

The sklearn.metrics implements several losses, scores and utility functions to measure classification performance.

Some of these are restricted to the binary classification case:

`auc_score`(y_true, y_score)	Compute Area Under the Curve (AUC) from prediction scores
`average_precision_score`(y_true, y_score)	Compute average precision (AP) from prediction scores
`hinge_loss`(y_true, pred_decision[, ...])	Average hinge loss (non-regularized)
`matthews_corrcoef`(y_true, y_pred)	Compute the Matthews correlation coefficient (MCC) for binary classes
`precision_recall_curve`(y_true, probas_pred)	Compute precision-recall pairs for different probability thresholds
`roc_curve`(y_true, y_score[, pos_label])	Compute Receiver operating characteristic (ROC)

Others also work in the multiclass case:

`accuracy_score`(y_true, y_pred)	Accuracy classification score
`classification_report`(y_true, y_pred[, ...])	Build a text report showing the main classification metrics
`confusion_matrix`(y_true, y_pred[, labels])	Compute confusion matrix to evaluate the accuracy of a classification
`f1_score`(y_true, y_pred[, labels, ...])	Compute the F1 score, also known as balanced F-score or F-measure
`fbeta_score`(y_true, y_pred, beta[, labels, ...])	Compute the F-beta score
`precision_recall_fscore_support`(y_true, y_pred)	Compute precision, recall, F-measure and support for each class
`precision_score`(y_true, y_pred[, labels, ...])	Compute the precision
`recall_score`(y_true, y_pred[, labels, ...])	Compute the recall
`zero_one_loss`(y_true, y_pred[, normalize])	Zero-One classification loss

Some metrics might require probability estimates of the positive class, confidence values or binary decisions value.

In the following sub-sections, we will describe each of those functions.

5.5.1.1. Accuracy score¶

The accuracy_score function computes the accuracy, the fraction of correct predictions.

If $\hat{y}_i$ is the predicted value of the $i$ -th sample and $y_i$ is the corresponding true value, then the fraction of correct predictions over $n_\text{samples}$ is defined as

$\texttt{accuracy}(y, \hat{y}) = \frac{1}{n_\text{samples}} \sum_{i=0}^{n_\text{samples}-1} 1(\hat{y} = y)$

where $1(x)$ is the indicator function.

>>> from sklearn.metrics import accuracy_score
>>> y_pred = [0, 2, 1, 3]
>>> y_true = [0, 1, 2, 3]
>>> accuracy_score(y_true, y_pred)
0.5

Example:

See Test with permutations the significance of a classification score for an example of accuracy score usage using permutations of the dataset.

5.5.1.2. Area under the curve (AUC)¶

The auc_score function computes the ‘area under the curve’ (AUC) which is the area under the receiver operating characteristic (ROC) curve.

This function requires the true binary value and the target scores, which can either be probability estimates of the positive class, confidence values, or binary decisions.

>>> import numpy as np
>>> from sklearn.metrics import auc_score
>>> y_true = np.array([0, 0, 1, 1])
>>> y_scores = np.array([0.1, 0.4, 0.35, 0.8])
>>> auc_score(y_true, y_scores)
0.75

For more information see the Wikipedia article on AUC and the Receiver operating characteristic (ROC) section.

5.5.1.3. Average precision score¶

The average_precision_score function computes the average precision (AP) from prediction scores. This score corresponds to the area under the precision-recall curve.

>>> import numpy as np
>>> from sklearn.metrics import average_precision_score
>>> y_true = np.array([0, 0, 1, 1])
>>> y_scores = np.array([0.1, 0.4, 0.35, 0.8])
>>> average_precision_score(y_true, y_scores)  
0.79...

For more information see the Wikipedia article on average precision and the Precision, recall and F-measures section.

5.5.1.4. Confusion matrix¶

The confusion_matrix function computes the confusion matrix to evaluate the accuracy on a classification problem.

By definition, a confusion matrix $C$ is such that $C_{i, j}$ is equal to the number of observations known to be in group $i$ but predicted to be in group $j$ . Here an example of such confusion matrix:

>>> from sklearn.metrics import confusion_matrix
>>> y_true = [2, 0, 2, 2, 0, 1]
>>> y_pred = [0, 0, 2, 2, 0, 2]
>>> confusion_matrix(y_true, y_pred)
array([[2, 0, 0],
       [0, 0, 1],
       [1, 0, 2]])

Here a visual representation of such confusion matrix (this figure comes from the Confusion matrix example):

Example:

See Confusion matrix for an example of confusion matrix usage to evaluate the quality of the output of a classifier.
See Recognizing hand-written digits for an example of confusion matrix usage in the classification of hand-written digits.
See Classification of text documents using sparse features for an example of confusion matrix usage in the classification of text documents.

5.5.1.5. Classification report¶

The classification_report function builds a text report showing the main classification metrics. Here a small example with custom target_names and infered labels:

>>> from sklearn.metrics import classification_report
>>> y_true = [0, 1, 2, 2, 0]
>>> y_pred = [0, 0, 2, 2, 0]
>>> target_names = ['class 0', 'class 1', 'class 2']
>>> print(classification_report(y_true, y_pred, target_names=target_names))
             precision    recall  f1-score   support

    class 0       0.67      1.00      0.80         2
    class 1       0.00      0.00      0.00         1
    class 2       1.00      1.00      1.00         2

avg / total       0.67      0.80      0.72         5

Example:

See Recognizing hand-written digits for an example of classification report usage in the classification of the hand-written digits.
See Classification of text documents using sparse features for an example of classification report usage in the classification of text documents.
See Parameter estimation using grid search with a nested cross-validation for an example of classification report usage in parameter estimation using grid search with a nested cross-validation.

5.5.1.6. Precision, recall and F-measures¶

The precision is intuitively the ability of the classifier not to label as positive a sample that is negative.

The recall is intuitively the ability of the classifier to find all the positive samples.

The F-measure ( $F_\beta$ and $F_1$ measures) can be interpreted as a weighted harmonic mean of the precision and recall. A $F_\beta$ measure reaches its best value at 1 and worst score at 0. With $\beta = 1$ , the $F_\beta$ measure leads to the $F_1$ measure, wheres the recall and the precsion are equally important.

Several functions allow you to analyze the precision, recall and F-measures score:

`f1_score`(y_true, y_pred[, labels, ...])	Compute the F1 score, also known as balanced F-score or F-measure
`fbeta_score`(y_true, y_pred, beta[, labels, ...])	Compute the F-beta score
`precision_recall_curve`(y_true, probas_pred)	Compute precision-recall pairs for different probability thresholds
`precision_recall_fscore_support`(y_true, y_pred)	Compute precision, recall, F-measure and support for each class
`precision_score`(y_true, y_pred[, labels, ...])	Compute the precision
`recall_score`(y_true, y_pred[, labels, ...])	Compute the recall

Note that the precision_recall_curve function is restricted to the binary case.

The average precision score might also interest you. See the Average precision score section.

Examples:

See Classification of text documents using sparse features for an example of f1_score usage with classification of text documents.
See Parameter estimation using grid search with a nested cross-validation for an example of precision_score and recall_score usage in parameter estimation using grid search with a nested cross-validation.
See Precision-Recall for an example of precision-Recall metric to evaluate the quality of the output of a classifier with precision_recall_curve.
See Sparse recovery: feature selection for sparse linear models for an example of precision_recall_curve usage in feature selection for sparse linear models.

5.5.1.6.1. Binary classification¶

In a binary classification task, the terms ‘’positive’’ and ‘’negative’’ refer to the classifier’s prediction and the terms ‘’true’’ and ‘’false’’ refer to whether that prediction corresponds to the external judgment (sometimes known as the ‘’observation’‘). Given these definitions, we can formulate the following table:

	Actual class (observation)
Predicted class (expectation)	tp (true positive) Correct result	fp (false positive) Unexpected result
Predicted class (expectation)	fn (false negative) Missing result	tn (true negative) Correct absence of result

In this context, we can define the notions of precision, recall and F-measure:

$\text{precision} = \frac{tp}{tp + fp},$

$\text{recall} = \frac{tp}{tp + fn},$

$F_\beta = (1 + \beta^2) \frac{\text{precision} \times \text{recall}}{\beta^2 \text{precision} + \text{recall}}.$

Here some small examples in binary classification:

>>> from sklearn import metrics
>>> y_pred = [0, 1, 0, 0]
>>> y_true = [0, 1, 0, 1]
>>> metrics.precision_score(y_true, y_pred)
1.0
>>> metrics.recall_score(y_true, y_pred)
0.5
>>> metrics.f1_score(y_true, y_pred)  
0.66...
>>> metrics.fbeta_score(y_true, y_pred, beta=0.5)  
0.83...
>>> metrics.fbeta_score(y_true, y_pred, beta=1)  
0.66...
>>> metrics.fbeta_score(y_true, y_pred, beta=2) 
0.55...
>>> metrics.precision_recall_fscore_support(y_true, y_pred, beta=0.5)  
(array([ 0.66...,  1.        ]), array([ 1. ,  0.5]), array([ 0.71...,  0.83...]), array([2, 2]...))


>>> import numpy as np
>>> from sklearn.metrics import precision_recall_curve
>>> y_true = np.array([0, 0, 1, 1])
>>> y_scores = np.array([0.1, 0.4, 0.35, 0.8])
>>> precision, recall, threshold = precision_recall_curve(y_true, y_scores)
>>> precision  
array([ 0.66...,  0.5       ,  1.        ,  1.        ])
>>> recall
array([ 1. ,  0.5,  0.5,  0. ])
>>> threshold
array([ 0.35,  0.4 ,  0.8 ])

5.5.1.6.2. Multiclass and multilabels classification¶

In multiclass and multilabels classification task, the notions of precision, recall and F-measures can be applied to each label independently.

Moreover, these notions can be further extended. The functions f1_score, fbeta_score, precision_recall_fscore_support, precision_score and recall_score support an argument called average which defines the type of averaging:

"macro": average over classes (does not take imbalance into account).

"micro": average over instances (takes imbalance into account).

"weighted": average weighted by support (takes imbalance into account). It can result in F1 score that is not between precision and recall.

None: no averaging is performed.

Warning

Currently those functions support only the multiclass case. However the following definitions are general and remain valid in the multilabel case.

Let’s define some notations:

$n_\text{labels}$ and $n_\text{samples}$ denotes respectively the number of labels and the number of samples.

$\texttt{precision}_j$ , $\texttt{recall}_j$ and ${F_\beta}_j$ are respectively the precision, the recall and $F_\beta$ measure for the $j$ -th label;

$tp_j$ , $fp_j$ and $fn_j$ respectively the number of true positives, false positives and false negatives for the $j$ -th label;

$y_i$ is the set of true label and $\hat{y}_i$ is the set of predicted for the $i$ -th sample;

The macro precision, recall and $F_\beta$ are averaged over all labels

$\texttt{macro\_{}precision} = \frac{1}{n_\text{labels}} \sum_{j=0}^{n_\text{labels} - 1} \texttt{precision}_j,$

$\texttt{macro\_{}recall} = \frac{1}{n_\text{labels}} \sum_{j=0}^{n_\text{labels} - 1} \texttt{recall}_j,$

$\texttt{macro\_{}F\_{}beta} = \frac{1}{n_\text{labels}} \sum_{j=0}^{n_\text{labels} - 1} {F_\beta}_j.$

The micro precision, recall and $F_\beta$ are averaged over all instances

$\texttt{micro\_{}precision} = \frac{\sum_{j=0}^{n_\text{labels} - 1} tp_j}{\sum_{j=0}^{n_\text{labels} - 1} tp_j + \sum_{j=0}^{n_\text{labels} - 1} fp_j},$

$\texttt{micro\_{}recall} = \frac{\sum_{j=0}^{n_\text{labels} - 1} tp_j}{\sum_{j=0}^{n_\text{labels} - 1} tp_j + \sum_{j=0}^{n_\text{labels} - 1} fn_j},$

$\texttt{micro\_{}F\_{}beta} = (1 + \beta^2) \frac{\texttt{micro\_{}precision} \times \texttt{micro\_{}recall}}{\beta^2 \texttt{micro\_{}precision} + \texttt{micro\_{}recall}}.$

The weighted precision, recall and $F_\beta$ are averaged weighted by their support

$\texttt{weighted\_{}precision}(y,\hat{y}) &= \frac{1}{n_\text{samples}} \sum_{i=0}^{n_\text{samples} - 1} \frac{|y_i \cap \hat{y}_i|}{|y_i|},$

$\texttt{weighted\_{}recall}(y,\hat{y}) &= \frac{1}{n_\text{samples}} \sum_{i=0}^{n_\text{samples} - 1} \frac{|y_i \cap \hat{y}_i|}{|\hat{y}_i|},$

$\texttt{weighted\_{}F\_{}beta}(y,\hat{y}) &= \frac{1}{n_\text{samples}} \sum_{i=0}^{n_\text{samples} - 1} (1 + \beta^2)\frac{|y_i \cap \hat{y}_i|}{\beta^2 |\hat{y}_i| + |y_i|}.$

Here an example where average is set to average to macro:

>>> from sklearn import metrics
>>> y_true = [0, 1, 2, 0, 1, 2]
>>> y_pred = [0, 2, 1, 0, 0, 1]
>>> metrics.precision_score(y_true, y_pred, average='macro')  
0.22...
>>> metrics.recall_score(y_true, y_pred, average='macro')  
0.33...
>>> metrics.fbeta_score(y_true, y_pred, average='macro', beta=0.5)  
0.23...
>>> metrics.f1_score(y_true, y_pred, average='macro')  
0.26...
>>> metrics.precision_recall_fscore_support(y_true, y_pred, average='macro')  
(0.22..., 0.33..., 0.26..., None)

Here an example where average is set to to micro:

>>> from sklearn import metrics
>>> y_true = [0, 1, 2, 0, 1, 2]
>>> y_pred = [0, 2, 1, 0, 0, 1]
>>> metrics.precision_score(y_true, y_pred, average='micro')  
0.33...
>>> metrics.recall_score(y_true, y_pred, average='micro')  
0.33...
>>> metrics.f1_score(y_true, y_pred, average='micro')  
0.33...
>>> metrics.fbeta_score(y_true, y_pred, average='micro', beta=0.5)  
0.33...
>>> metrics.precision_recall_fscore_support(y_true, y_pred, average='micro')  
(0.33..., 0.33..., 0.33..., None)

Here an example where average is set to to weighted:

>>> from sklearn import metrics
>>> y_true = [0, 1, 2, 0, 1, 2]
>>> y_pred = [0, 2, 1, 0, 0, 1]
>>> metrics.precision_score(y_true, y_pred, average='weighted')  
0.22...
>>> metrics.recall_score(y_true, y_pred, average='weighted')  
0.33...
>>> metrics.fbeta_score(y_true, y_pred, average='weighted', beta=0.5)  
0.23...
>>> metrics.f1_score(y_true, y_pred, average='weighted')  
0.26...
>>> metrics.precision_recall_fscore_support(y_true, y_pred, average='weighted')  
(0.22..., 0.33..., 0.26..., None)

Here an example where average is set to None:

>>> from sklearn import metrics
>>> y_true = [0, 1, 2, 0, 1, 2]
>>> y_pred = [0, 2, 1, 0, 0, 1]
>>> metrics.precision_score(y_true, y_pred, average=None)  
array([ 0.66...,  0.        ,  0.        ])
>>> metrics.recall_score(y_true, y_pred, average=None)
array([ 1.,  0.,  0.])
>>> metrics.f1_score(y_true, y_pred, average=None)  
array([ 0.8,  0. ,  0. ])
>>> metrics.fbeta_score(y_true, y_pred, average=None, beta=0.5)  
array([ 0.71...,  0.        ,  0.        ])
>>> metrics.precision_recall_fscore_support(y_true, y_pred, beta=0.5)  
(array([ 0.66...,  0.        ,  0.        ]), array([ 1.,  0.,  0.]), array([ 0.71...,  0.        ,  0.        ]), array([2, 2, 2]...))

5.5.1.7. Hinge loss¶

The hinge_loss function computes the average hinge loss function. The hinge loss is used in maximal margin classification as support vector machines.

If the labels are encoded with +1 and -1, $y$ : is the true value and $w$ is the predicted decisions as output by decision_function, then the hinge loss is defined as:

$L_\text{Hinge}(y, w) = \max\left\{1 - wy, 0\right\} = \left|1 - wy\right|_+$

Here a small example demonstrating the use of the hinge_loss function with a svm classifier:

>>> from sklearn import svm
>>> from sklearn.metrics import hinge_loss
>>> X = [[0], [1]]
>>> y = [-1, 1]
>>> est = svm.LinearSVC(random_state=0)
>>> est.fit(X, y)
LinearSVC(C=1.0, class_weight=None, dual=True, fit_intercept=True,
     intercept_scaling=1, loss='l2', multi_class='ovr', penalty='l2',
     random_state=0, tol=0.0001, verbose=0)
>>> pred_decision = est.decision_function([[-2], [3], [0.5]])
>>> pred_decision  
array([-2.18...,  2.36...,  0.09...])
>>> hinge_loss([-1, 1, 1], pred_decision)  
0.30...

5.5.1.8. Matthews correlation coefficient¶

The matthews_corrcoef function computes the Matthew’s correlation coefficient (MCC) for binary classes (quoting the Wikipedia article on the Matthew’s correlation coefficient):

“The Matthews correlation coefficient is used in machine learning as a measure of the quality of binary (two-class) classifications. It takes into account true and false positives and negatives and is generally regarded as a balanced measure which can be used even if the classes are of very different sizes. The MCC is in essence a correlation coefficient value between -1 and +1. A coefficient of +1 represents a perfect prediction, 0 an average random prediction and -1 an inverse prediction. The statistic is also known as the phi coefficient.”

If $tp$ , $tn$ , $fp$ and $fn$ are respectively the number of true positives, true negatives, false positives ans false negatives, the MCC coefficient is defined as

$MCC = \frac{tp \times tn - fp \times fn}{\sqrt{(tp + fp)(tp + fn)(tn + fp)(tn + fn)}}.$

Here a small example illustrating the usage of the matthews_corrcoef function:

>>> from sklearn.metrics import matthews_corrcoef
>>> y_true = [+1, +1, +1, -1]
>>> y_pred = [+1, -1, +1, +1]
>>> matthews_corrcoef(y_true, y_pred)  
-0.33...

5.5.1.9. Receiver operating characteristic (ROC)¶

The function roc_curve computes the receiver operating characteristic curve, or ROC curve (quoting Wikipedia):

“A receiver operating characteristic (ROC), or simply ROC curve, is a graphical plot which illustrates the performance of a binary classifier system as its discrimination threshold is varied. It is created by plotting the fraction of true positives out of the positives (TPR = true positive rate) vs. the fraction of false positives out of the negatives (FPR = false positive rate), at various threshold settings. TPR is also known as sensitivity, and FPR is one minus the specificity or true negative rate.”

Here a small example of how to use the roc_curve function:

>>> import numpy as np
>>> from sklearn import metrics
>>> y = np.array([1, 1, 2, 2])
>>> scores = np.array([0.1, 0.4, 0.35, 0.8])
>>> fpr, tpr, thresholds = metrics.roc_curve(y, scores, pos_label=2)
>>> fpr
array([ 0. ,  0.5,  0.5,  1. ])

The following figure shows an example of such ROC curve.

Examples:

See Receiver operating characteristic (ROC) for an example of receiver operating characteristic (ROC) metric to evaluate the quality of the output of a classifier.
See Receiver operating characteristic (ROC) with cross validation for an example of receiver operating characteristic (ROC) metric to evaluate the quality of the output of a classifier using cross-validation.
See Species distribution modeling for an example of receiver operating characteristic (ROC) metric to model species distribution.

5.5.1.10. Zero one loss¶

The zero_one_loss function computes the sum or the average of the 0-1 classification loss ( $L_{0-1}$ ) over $n_{\text{samples}}$ . By defaults, the function normalizes over the sample. To get the sum of the $L_{0-1}$ , set normalize to False.

If $\hat{y}_i$ is the predicted value of the $i$ -th sample and $y_i$ is the corresponding true value, then the 0-1 loss $L_{0-1}$ is defined as:

$L_{0-1}(y_i, \hat{y}_i) = 1(\hat{y} \not= y)$

where $1(x)$ is the indicator function.

>>> from sklearn.metrics import zero_one_loss
>>> y_pred = [1, 2, 3, 4]
>>> y_true = [2, 2, 3, 4]

G

>>> zero_one_loss(y_true, y_pred)
0.25
>>> zero_one_loss(y_true, y_pred, normalize=False)
1

Example:

See Recursive feature elimination with cross-validation for an example of the zero one loss usage to perform recursive feature elimination with cross-validation.

5.5.2. Regression metrics¶

The sklearn.metrics implements several losses, scores and utility functions to measure regression performance. Some of those have been enhanced to handle the multioutput case: mean_absolute_error, mean_absolute_error and r2_score.

5.5.2.1. Explained variance score¶

The explained_variance_score computes the explained variance regression score.

If $\hat{y}$ is the estimated target output and $y$ is the corresponding (correct) target output, then the explained variance is estimated as follow:

$\texttt{explained\_{}variance}(y, \hat{y}) = 1 - \frac{Var\{ y - \hat{y}\}}{Var\{y\}}$

The best possible score is 1.0, lower values are worse.

Here a small example of usage of the explained_variance_score function:

>>> from sklearn.metrics import explained_variance_score
>>> y_true = [3, -0.5, 2, 7]
>>> y_pred = [2.5, 0.0, 2, 8]
>>> explained_variance_score(y_true, y_pred)  
0.957...

5.5.2.2. Mean absolute error¶

The mean_absolute_error function computes the mean absolute error, which is a risk function corresponding to the expected value of the absolute error loss or $l1$ -norm loss.

If $\hat{y}_i$ is the predicted value of the $i$ -th sample and $y_i$ is the corresponding true value, then the mean absolute error (MAE) estimated over $n_{\text{samples}}$ is defined as

$\text{MAE}(y, \hat{y}) = \frac{1}{n_{\text{samples}}} \sum_{i=0}^{n_{\text{samples}}-1} \left| y_i - \hat{y}_i \right|.$

Here a small example of usage of the mean_absolute_error function:

>>> from sklearn.metrics import mean_absolute_error
>>> y_true = [3, -0.5, 2, 7]
>>> y_pred = [2.5, 0.0, 2, 8]
>>> mean_absolute_error(y_true, y_pred)
0.5
>>> y_true = [[0.5, 1], [-1, 1], [7, -6]]
>>> y_pred = [[0, 2], [-1, 2], [8, -5]]
>>> mean_absolute_error(y_true, y_pred)
0.75

5.5.2.3. Mean squared error¶

The mean_squared_error function computes the mean square error, which is a risk function corresponding to the expected value of the squared error loss or quadratic loss.

If $\hat{y}_i$ is the predicted value of the $i$ -th sample and $y_i$ is the corresponding true value, then the mean squared error (MSE) estimated over $n_{\text{samples}}$ is defined as

$\text{MSE}(y, \hat{y}) = \frac{1}{n_\text{samples}} \sum_{i=0}^{n_\text{samples} - 1} (y_i - \hat{y}_i)^2.$

Here a small example of usage of the mean_squared_error function:

>>> from sklearn.metrics import mean_squared_error
>>> y_true = [3, -0.5, 2, 7]
>>> y_pred = [2.5, 0.0, 2, 8]
>>> mean_squared_error(y_true, y_pred)
0.375
>>> y_true = [[0.5, 1], [-1, 1], [7, -6]]
>>> y_pred = [[0, 2], [-1, 2], [8, -5]]
>>> mean_squared_error(y_true, y_pred)  
0.7083...

Examples:

See Gradient Boosting regression for an example of mean squared error usage to evaluate gradient boosting regression.

5.5.2.4. R² score, the coefficient of determination¶

The r2_score function computes R², the coefficient of determination. It provides a measure of how well future samples are likely to be predicted by the model.

If $\hat{y}_i$ is the predicted value of the $i$ -th sample and $y_i$ is the corresponding true value, then the score R² estimated over $n_{\text{samples}}$ is defined as

$R^2(y, \hat{y}) = 1 - \frac{\sum_{i=0}^{n_{\text{samples}} - 1} (y_i - \hat{y}_i)^2}{\sum_{i=0}^{n_\text{samples} - 1} (y_i - \bar{y})^2}$

where $\bar{y} = \frac{1}{n_{\text{samples}}} \sum_{i=0}^{n_{\text{samples}} - 1} y_i$ .

Here a small example of usage of the r2_score function:

>>> from sklearn.metrics import r2_score
>>> y_true = [3, -0.5, 2, 7]
>>> y_pred = [2.5, 0.0, 2, 8]
>>> r2_score(y_true, y_pred)  
0.948...
>>> y_true = [[0.5, 1], [-1, 1], [7, -6]]
>>> y_pred = [[0, 2], [-1, 2], [8, -5]]
>>> r2_score(y_true, y_pred)  
0.938...

Example:

See Lasso and Elastic Net for Sparse Signals for an example of R² score usage to evaluate Lasso and Elastic Net on sparse signals.

5.5.3. Clustering metrics¶

The sklearn.metrics implements several losses, scores and utility function for more information see the Clustering performance evaluation section.

5.5.4. Dummy estimators¶

When doing supervised learning, a simple sanity check consists in comparing one’s estimator against simple rules of thumb. DummyClassifier implements three such simple strategies for classification:

stratified generates randomly predictions by respecting the training set’s class distribution,
most_frequent always predicts the most frequent label in the training set,
uniform generates predictions uniformly at random.

Note that with all these strategies, the predict method completely ignores the input data!

To illustrate DummyClassifier, first let’s create an imbalanced dataset:

>>> from sklearn.datasets import load_iris
>>> from sklearn.cross_validation import train_test_split
>>> iris = load_iris()
>>> X, y = iris.data, iris.target
>>> y[y != 1] = -1
>>> X_train, X_test, y_train, y_test = train_test_split(X, y, random_state=0)

Next, let’s compare the accuracy of SVC and most_frequent:

>>> from sklearn.dummy import DummyClassifier
>>> from sklearn.svm import SVC
>>> clf = SVC(kernel='linear', C=1).fit(X_train, y_train)
>>> clf.score(X_test, y_test) 
0.63...
>>> clf = DummyClassifier(strategy='most_frequent',random_state=0)
>>> clf.fit(X_train, y_train)
DummyClassifier(random_state=0, strategy='most_frequent')
>>> clf.score(X_test, y_test)  
0.57...

We see that SVC doesn’t do much better than a dummy classifier. Now, let’s change the kernel:

>>> clf = SVC(kernel='rbf', C=1).fit(X_train, y_train)
>>> clf.score(X_test, y_test)  
0.97...

We see that the accuracy was boosted to almost 100%. For a better estimate of the accuracy, it is recommended to use a cross validation strategy, if it is not too CPU costly. For more information see the Cross-Validation: evaluating estimator performance section. Moreover if you want to optimize over the parameter space, it is highly recommended to use an appropriate methodology see the Grid Search: setting estimator parameters section.

More generally, when the accuracy of a classifier is too close to random classification, it probably means that something went wrong: features are not helpful, a hyper parameter is not correctly tuned, the classifier is suffering from class imbalance, etc...

DummyRegressor implements a simple rule of thumb for regression: always predict the mean of the training targets.

Citing

This page

5.5. Model evaluation¶

5.5.1. Classification metrics¶

5.5.1.1. Accuracy score¶

5.5.1.2. Area under the curve (AUC)¶

5.5.1.3. Average precision score¶

5.5.1.4. Confusion matrix¶

5.5.1.5. Classification report¶

5.5.1.6. Precision, recall and F-measures¶

5.5.1.6.1. Binary classification¶

5.5.1.6.2. Multiclass and multilabels classification¶

5.5.1.7. Hinge loss¶

5.5.1.8. Matthews correlation coefficient¶

5.5.1.9. Receiver operating characteristic (ROC)¶

5.5.1.10. Zero one loss¶

5.5.2. Regression metrics¶

5.5.2.1. Explained variance score¶

5.5.2.2. Mean absolute error¶

5.5.2.3. Mean squared error¶

5.5.2.4. R² score, the coefficient of determination¶

5.5.3. Clustering metrics¶

5.5.4. Dummy estimators¶