sklearn.cross_validation.permutation_test_score¶
- sklearn.cross_validation.permutation_test_score(estimator, X, y, cv=None, n_permutations=100, n_jobs=1, labels=None, random_state=0, verbose=0, scoring=None)[source]¶
Evaluate the significance of a cross-validated score with permutations
Parameters: estimator : estimator object implementing ‘fit’
The object to use to fit the data.
X : array-like of shape at least 2D
The data to fit.
y : array-like
The target variable to try to predict in the case of supervised learning.
scoring : string, callable or None, optional, default: None
A string (see model evaluation documentation) or a scorer callable object / function with signature scorer(estimator, X, y).
cv : integer or cross-validation generator, optional
If an integer is passed, it is the number of fold (default 3). Specific cross-validation objects can be passed, see sklearn.cross_validation module for the list of possible objects.
n_permutations : integer, optional
Number of times to permute y.
n_jobs : integer, optional
The number of CPUs to use to do the computation. -1 means ‘all CPUs’.
labels : array-like of shape [n_samples] (optional)
Labels constrain the permutation among groups of samples with a same label.
random_state : RandomState or an int seed (0 by default)
A random number generator instance to define the state of the random permutations generator.
verbose : integer, optional
The verbosity level.
Returns: score : float
The true score without permuting targets.
permutation_scores : array, shape (n_permutations,)
The scores obtained for each permutations.
pvalue : float
The returned value equals p-value if scoring returns bigger numbers for better scores (e.g., accuracy_score). If scoring is rather a loss function (i.e. when lower is better such as with mean_squared_error) then this is actually the complement of the p-value: 1 - p-value.
Notes
This function implements Test 1 in:
Ojala and Garriga. Permutation Tests for Studying Classifier Performance. The Journal of Machine Learning Research (2010) vol. 11
