User Guide

• 1. Supervised learning
  • 1.1. Generalized Linear Models
    • 1.1.1. Ordinary Least Squares
      • 1.1.1.1. Ordinary Least Squares Complexity
    • 1.1.2. Ridge Regression
      • 1.1.2.1. Ridge Complexity
      • 1.1.2.2. Setting the regularization parameter: generalized Cross-Validation
    • 1.1.3. Lasso
      • 1.1.3.1. Setting regularization parameter
        • 1.1.3.1.1. Using cross-validation
        • 1.1.3.1.2. Information-criteria based model selection
    • 1.1.4. Elastic Net
    • 1.1.5. Multi-task Lasso
    • 1.1.6. Least Angle Regression
    • 1.1.7. LARS Lasso
      • 1.1.7.1. Mathematical formulation
    • 1.1.8. Orthogonal Matching Pursuit (OMP)
    • 1.1.9. Bayesian Regression
      • 1.1.9.1. Bayesian Ridge Regression
      • 1.1.9.2. Automatic Relevance Determination - ARD
    • 1.1.10. Logistic regression
    • 1.1.11. Stochastic Gradient Descent - SGD
    • 1.1.12. Perceptron
    • 1.1.13. Passive Aggressive Algorithms
    • 1.1.14. Robustness regression: outliers and modeling errors
      • 1.1.14.1. Different scenario and useful concepts
      • 1.1.14.2. RANSAC: RANdom SAmple Consensus
        • 1.1.14.2.1. Details of the algorithm
      • 1.1.14.3. Theil-Sen estimator: generalized-median-based estimator
        • 1.1.14.3.1. Theoretical considerations
    • 1.1.15. Polynomial regression: extending linear models with basis functions
  • 1.2. Linear and quadratic discriminant analysis
    • 1.2.1. Dimensionality reduction using LDA
    • 1.2.2. Mathematical Idea
    • 1.2.3. Shrinkage
    • 1.2.4. Estimation algorithms
  • 1.3. Kernel ridge regression
  • 1.4. Support Vector Machines
    • 1.4.1. Classification
      • 1.4.1.1. Multi-class classification
      • 1.4.1.2. Scores and probabilities
      • 1.4.1.3. Unbalanced problems
    • 1.4.2. Regression
    • 1.4.3. Density estimation, novelty detection
    • 1.4.4. Complexity
    • 1.4.5. Tips on Practical Use
    • 1.4.6. Kernel functions
      • 1.4.6.1. Custom Kernels
        • 1.4.6.1.1. Using Python functions as kernels
        • 1.4.6.1.2. Using the Gram matrix
        • 1.4.6.1.3. Parameters of the RBF Kernel
    • 1.4.7. Mathematical formulation
      • 1.4.7.1. SVC
      • 1.4.7.2. NuSVC
      • 1.4.7.3. SVR
    • 1.4.8. Implementation details
  • 1.5. Stochastic Gradient Descent
    • 1.5.1. Classification
    • 1.5.2. Regression
    • 1.5.3. Stochastic Gradient Descent for sparse data
    • 1.5.4. Complexity
    • 1.5.5. Tips on Practical Use
    • 1.5.6. Mathematical formulation
      • 1.5.6.1. SGD
    • 1.5.7. Implementation details
  • 1.6. Nearest Neighbors
    • 1.6.1. Unsupervised Nearest Neighbors
      • 1.6.1.1. Finding the Nearest Neighbors
      • 1.6.1.2. KDTree and BallTree Classes
    • 1.6.2. Nearest Neighbors Classification
    • 1.6.3. Nearest Neighbors Regression
    • 1.6.4. Nearest Neighbor Algorithms
      • 1.6.4.1. Brute Force
      • 1.6.4.2. K-D Tree
      • 1.6.4.3. Ball Tree
      • 1.6.4.4. Choice of Nearest Neighbors Algorithm
      • 1.6.4.5. Effect of leaf_size
    • 1.6.5. Nearest Centroid Classifier
      • 1.6.5.1. Nearest Shrunken Centroid
    • 1.6.6. Approximate Nearest Neighbors
      • 1.6.6.1. Locality Sensitive Hashing Forest
      • 1.6.6.2. Mathematical description of Locality Sensitive Hashing
  • 1.7. Gaussian Processes
    • 1.7.1. Examples
      • 1.7.1.1. An introductory regression example
      • 1.7.1.2. Fitting Noisy Data
    • 1.7.2. Mathematical formulation
      • 1.7.2.1. The initial assumption
      • 1.7.2.2. The best linear unbiased prediction (BLUP)
      • 1.7.2.3. The empirical best linear unbiased predictor (EBLUP)
    • 1.7.3. Correlation Models
    • 1.7.4. Regression Models
    • 1.7.5. Implementation details
  • 1.8. Cross decomposition
  • 1.9. Naive Bayes
    • 1.9.1. Gaussian Naive Bayes
    • 1.9.2. Multinomial Naive Bayes
    • 1.9.3. Bernoulli Naive Bayes
    • 1.9.4. Out-of-core naive Bayes model fitting
  • 1.10. Decision Trees
    • 1.10.1. Classification
    • 1.10.2. Regression
    • 1.10.3. Multi-output problems
    • 1.10.4. Complexity
    • 1.10.5. Tips on practical use
    • 1.10.6. Tree algorithms: ID3, C4.5, C5.0 and CART
    • 1.10.7. Mathematical formulation
      • 1.10.7.1. Classification criteria
      • 1.10.7.2. Regression criteria
  • 1.11. Ensemble methods
    • 1.11.1. Bagging meta-estimator
    • 1.11.2. Forests of randomized trees
      • 1.11.2.1. Random Forests
      • 1.11.2.2. Extremely Randomized Trees
      • 1.11.2.3. Parameters
      • 1.11.2.4. Parallelization
      • 1.11.2.5. Feature importance evaluation
      • 1.11.2.6. Totally Random Trees Embedding
    • 1.11.3. AdaBoost
      • 1.11.3.1. Usage
    • 1.11.4. Gradient Tree Boosting
      • 1.11.4.1. Classification
      • 1.11.4.2. Regression
      • 1.11.4.3. Fitting additional weak-learners
      • 1.11.4.4. Controlling the tree size
      • 1.11.4.5. Mathematical formulation
        • 1.11.4.5.1. Loss Functions
      • 1.11.4.6. Regularization
        • 1.11.4.6.1. Shrinkage
        • 1.11.4.6.2. Subsampling
      • 1.11.4.7. Interpretation
        • 1.11.4.7.1. Feature importance
        • 1.11.4.7.2. Partial dependence
    • 1.11.5. VotingClassifier
      • 1.11.5.1. Majority Class Labels (Majority/Hard Voting)
        • 1.11.5.1.1. Usage
      • 1.11.5.2. Weighted Average Probabilities (Soft Voting)
      • 1.11.5.3. Using the VotingClassifier with GridSearch
        • 1.11.5.3.1. Usage
  • 1.12. Multiclass and multilabel algorithms
    • 1.12.1. Multilabel classification format
    • 1.12.2. One-Vs-The-Rest
      • 1.12.2.1. Multiclass learning
      • 1.12.2.2. Multilabel learning
    • 1.12.3. One-Vs-One
      • 1.12.3.1. Multiclass learning
    • 1.12.4. Error-Correcting Output-Codes
      • 1.12.4.1. Multiclass learning
  • 1.13. Feature selection
    • 1.13.1. Removing features with low variance
    • 1.13.2. Univariate feature selection
    • 1.13.3. Recursive feature elimination
    • 1.13.4. L1-based feature selection
      • 1.13.4.1. Selecting non-zero coefficients
      • 1.13.4.2. Randomized sparse models
    • 1.13.5. Tree-based feature selection
    • 1.13.6. Feature selection as part of a pipeline
  • 1.14. Semi-Supervised
    • 1.14.1. Label Propagation
  • 1.15. Isotonic regression
  • 1.16. Probability calibration
• 2. Unsupervised learning
  • 2.1. Gaussian mixture models
    • 2.1.1. GMM classifier
      • 2.1.1.1. Pros and cons of class GMM: expectation-maximization inference
        • 2.1.1.1.1. Pros
        • 2.1.1.1.2. Cons
      • 2.1.1.2. Selecting the number of components in a classical GMM
      • 2.1.1.3. Estimation algorithm Expectation-maximization
    • 2.1.2. VBGMM classifier: variational Gaussian mixtures
      • 2.1.2.1. Pros and cons of class VBGMM: variational inference
        • 2.1.2.1.1. Pros
        • 2.1.2.1.2. Cons
      • 2.1.2.2. Estimation algorithm: variational inference
    • 2.1.3. DPGMM classifier: Infinite Gaussian mixtures
      • 2.1.3.1. Pros and cons of class DPGMM: Dirichlet process mixture model
        • 2.1.3.1.1. Pros
        • 2.1.3.1.2. Cons
      • 2.1.3.2. The Dirichlet Process
  • 2.2. Manifold learning
    • 2.2.1. Introduction
    • 2.2.2. Isomap
      • 2.2.2.1. Complexity
    • 2.2.3. Locally Linear Embedding
      • 2.2.3.1. Complexity
    • 2.2.4. Modified Locally Linear Embedding
      • 2.2.4.1. Complexity
    • 2.2.5. Hessian Eigenmapping
      • 2.2.5.1. Complexity
    • 2.2.6. Spectral Embedding
      • 2.2.6.1. Complexity
    • 2.2.7. Local Tangent Space Alignment
      • 2.2.7.1. Complexity
    • 2.2.8. Multi-dimensional Scaling (MDS)
      • 2.2.8.1. Metric MDS
      • 2.2.8.2. Nonmetric MDS
    • 2.2.9. t-distributed Stochastic Neighbor Embedding (t-SNE)
    • 2.2.10. Tips on practical use
  • 2.3. Clustering
    • 2.3.1. Overview of clustering methods
    • 2.3.2. K-means
      • 2.3.2.1. Mini Batch K-Means
    • 2.3.3. Affinity Propagation
    • 2.3.4. Mean Shift
    • 2.3.5. Spectral clustering
      • 2.3.5.1. Different label assignment strategies
    • 2.3.6. Hierarchical clustering
      • 2.3.6.1. Different linkage type: Ward, complete and average linkage
      • 2.3.6.2. Adding connectivity constraints
      • 2.3.6.3. Varying the metric
    • 2.3.7. DBSCAN
    • 2.3.8. Birch
    • 2.3.9. Clustering performance evaluation
      • 2.3.9.1. Adjusted Rand index
        • 2.3.9.1.1. Advantages
        • 2.3.9.1.2. Drawbacks
        • 2.3.9.1.3. Mathematical formulation
      • 2.3.9.2. Mutual Information based scores
        • 2.3.9.2.1. Advantages
        • 2.3.9.2.2. Drawbacks
        • 2.3.9.2.3. Mathematical formulation
      • 2.3.9.3. Homogeneity, completeness and V-measure
        • 2.3.9.3.1. Advantages
        • 2.3.9.3.2. Drawbacks
        • 2.3.9.3.3. Mathematical formulation
      • 2.3.9.4. Silhouette Coefficient
        • 2.3.9.4.1. Advantages
        • 2.3.9.4.2. Drawbacks
  • 2.4. Biclustering
    • 2.4.1. Spectral Co-Clustering
      • 2.4.1.1. Mathematical formulation
    • 2.4.2. Spectral Biclustering
      • 2.4.2.1. Mathematical formulation
    • 2.4.3. Biclustering evaluation
  • 2.5. Decomposing signals in components (matrix factorization problems)
    • 2.5.1. Principal component analysis (PCA)
      • 2.5.1.1. Exact PCA and probabilistic interpretation
      • 2.5.1.2. Incremental PCA
      • 2.5.1.3. Approximate PCA
      • 2.5.1.4. Kernel PCA
      • 2.5.1.5. Sparse principal components analysis (SparsePCA and MiniBatchSparsePCA)
    • 2.5.2. Truncated singular value decomposition and latent semantic analysis
    • 2.5.3. Dictionary Learning
      • 2.5.3.1. Sparse coding with a precomputed dictionary
      • 2.5.3.2. Generic dictionary learning
      • 2.5.3.3. Mini-batch dictionary learning
    • 2.5.4. Factor Analysis
    • 2.5.5. Independent component analysis (ICA)
    • 2.5.6. Non-negative matrix factorization (NMF or NNMF)
  • 2.6. Covariance estimation
    • 2.6.1. Empirical covariance
    • 2.6.2. Shrunk Covariance
      • 2.6.2.1. Basic shrinkage
      • 2.6.2.2. Ledoit-Wolf shrinkage
      • 2.6.2.3. Oracle Approximating Shrinkage
    • 2.6.3. Sparse inverse covariance
    • 2.6.4. Robust Covariance Estimation
      • 2.6.4.1. Minimum Covariance Determinant
  • 2.7. Novelty and Outlier Detection
    • 2.7.1. Novelty Detection
    • 2.7.2. Outlier Detection
      • 2.7.2.1. Fitting an elliptic envelope
      • 2.7.2.2. One-class SVM versus elliptic envelope
  • 2.8. Density Estimation
    • 2.8.1. Density Estimation: Histograms
    • 2.8.2. Kernel Density Estimation
  • 2.9. Neural network models (unsupervised)
    • 2.9.1. Restricted Boltzmann machines
      • 2.9.1.1. Graphical model and parametrization
      • 2.9.1.2. Bernoulli Restricted Boltzmann machines
      • 2.9.1.3. Stochastic Maximum Likelihood learning
• 3. Model selection and evaluation
  • 3.1. Cross-validation: evaluating estimator performance
    • 3.1.1. Computing cross-validated metrics
      • 3.1.1.1. Obtaining predictions by cross-validation
    • 3.1.2. Cross validation iterators
      • 3.1.2.1. K-fold
      • 3.1.2.2. Stratified k-fold
      • 3.1.2.3. Leave-One-Out - LOO
      • 3.1.2.4. Leave-P-Out - LPO
      • 3.1.2.5. Leave-One-Label-Out - LOLO
      • 3.1.2.6. Leave-P-Label-Out
      • 3.1.2.7. Random permutations cross-validation a.k.a. Shuffle & Split
      • 3.1.2.8. Predefined Fold-Splits / Validation-Sets
      • 3.1.2.9. See also
    • 3.1.3. A note on shuffling
    • 3.1.4. Cross validation and model selection
  • 3.2. Grid Search: Searching for estimator parameters
    • 3.2.1. Exhaustive Grid Search
    • 3.2.2. Randomized Parameter Optimization
    • 3.2.3. Tips for parameter search
      • 3.2.3.1. Specifying an objective metric
      • 3.2.3.2. Composite estimators and parameter spaces
      • 3.2.3.3. Model selection: development and evaluation
      • 3.2.3.4. Parallelism
      • 3.2.3.5. Robustness to failure
    • 3.2.4. Alternatives to brute force parameter search
      • 3.2.4.1. Model specific cross-validation
        • 3.2.4.1.1. sklearn.linear_model.ElasticNetCV
        • 3.2.4.1.2. sklearn.linear_model.LarsCV
        • 3.2.4.1.3. sklearn.linear_model.LassoCV
          • 3.2.4.1.3.1. Examples using sklearn.linear_model.LassoCV
        • 3.2.4.1.4. sklearn.linear_model.LassoLarsCV
          • 3.2.4.1.4.1. Examples using sklearn.linear_model.LassoLarsCV
        • 3.2.4.1.5. sklearn.linear_model.LogisticRegressionCV
        • 3.2.4.1.6. sklearn.linear_model.MultiTaskElasticNetCV
        • 3.2.4.1.7. sklearn.linear_model.MultiTaskLassoCV
        • 3.2.4.1.8. sklearn.linear_model.OrthogonalMatchingPursuitCV
          • 3.2.4.1.8.1. Examples using sklearn.linear_model.OrthogonalMatchingPursuitCV
        • 3.2.4.1.9. sklearn.linear_model.RidgeCV
          • 3.2.4.1.9.1. Examples using sklearn.linear_model.RidgeCV
        • 3.2.4.1.10. sklearn.linear_model.RidgeClassifierCV
      • 3.2.4.2. Information Criterion
        • 3.2.4.2.1. sklearn.linear_model.LassoLarsIC
          • 3.2.4.2.1.1. Examples using sklearn.linear_model.LassoLarsIC
      • 3.2.4.3. Out of Bag Estimates
        • 3.2.4.3.1. sklearn.ensemble.RandomForestClassifier
          • 3.2.4.3.1.1. Examples using sklearn.ensemble.RandomForestClassifier
        • 3.2.4.3.2. sklearn.ensemble.RandomForestRegressor
          • 3.2.4.3.2.1. Examples using sklearn.ensemble.RandomForestRegressor
        • 3.2.4.3.3. sklearn.ensemble.ExtraTreesClassifier
          • 3.2.4.3.3.1. Examples using sklearn.ensemble.ExtraTreesClassifier
        • 3.2.4.3.4. sklearn.ensemble.ExtraTreesRegressor
          • 3.2.4.3.4.1. Examples using sklearn.ensemble.ExtraTreesRegressor
        • 3.2.4.3.5. sklearn.ensemble.GradientBoostingClassifier
          • 3.2.4.3.5.1. Examples using sklearn.ensemble.GradientBoostingClassifier
        • 3.2.4.3.6. sklearn.ensemble.GradientBoostingRegressor
          • 3.2.4.3.6.1. Examples using sklearn.ensemble.GradientBoostingRegressor
  • 3.3. Model evaluation: quantifying the quality of predictions
    • 3.3.1. The scoring parameter: defining model evaluation rules
      • 3.3.1.1. Common cases: predefined values
      • 3.3.1.2. Defining your scoring strategy from metric functions
      • 3.3.1.3. Implementing your own scoring object
    • 3.3.2. Classification metrics
      • 3.3.2.1. From binary to multiclass and multilabel
      • 3.3.2.2. Accuracy score
      • 3.3.2.3. Confusion matrix
      • 3.3.2.4. Classification report
      • 3.3.2.5. Hamming loss
      • 3.3.2.6. Jaccard similarity coefficient score
      • 3.3.2.7. Precision, recall and F-measures
        • 3.3.2.7.1. Binary classification
        • 3.3.2.7.2. Multiclass and multilabel classification
      • 3.3.2.8. Hinge loss
      • 3.3.2.9. Log loss
      • 3.3.2.10. Matthews correlation coefficient
      • 3.3.2.11. Receiver operating characteristic (ROC)
      • 3.3.2.12. Zero one loss
    • 3.3.3. Multilabel ranking metrics
      • 3.3.3.1. Coverage error
      • 3.3.3.2. Label ranking average precision
      • 3.3.3.3. Ranking loss
    • 3.3.4. Regression metrics
      • 3.3.4.1. Explained variance score
      • 3.3.4.2. Mean absolute error
      • 3.3.4.3. Mean squared error
      • 3.3.4.4. Median absolute error
      • 3.3.4.5. R² score, the coefficient of determination
    • 3.3.5. Clustering metrics
    • 3.3.6. Dummy estimators
  • 3.4. Model persistence
    • 3.4.1. Persistence example
    • 3.4.2. Security & maintainability limitations
  • 3.5. Validation curves: plotting scores to evaluate models
    • 3.5.1. Validation curve
    • 3.5.2. Learning curve
• 4. Dataset transformations
  • 4.1. Pipeline and FeatureUnion: combining estimators
    • 4.1.1. Pipeline: chaining estimators
      • 4.1.1.1. Usage
      • 4.1.1.2. Notes
    • 4.1.2. FeatureUnion: composite feature spaces
      • 4.1.2.1. Usage
  • 4.2. Feature extraction
    • 4.2.1. Loading features from dicts
    • 4.2.2. Feature hashing
      • 4.2.2.1. Implementation details
    • 4.2.3. Text feature extraction
      • 4.2.3.1. The Bag of Words representation
      • 4.2.3.2. Sparsity
      • 4.2.3.3. Common Vectorizer usage
      • 4.2.3.4. Tf–idf term weighting
      • 4.2.3.5. Decoding text files
      • 4.2.3.6. Applications and examples
      • 4.2.3.7. Limitations of the Bag of Words representation
      • 4.2.3.8. Vectorizing a large text corpus with the hashing trick
      • 4.2.3.9. Performing out-of-core scaling with HashingVectorizer
      • 4.2.3.10. Customizing the vectorizer classes
    • 4.2.4. Image feature extraction
      • 4.2.4.1. Patch extraction
      • 4.2.4.2. Connectivity graph of an image
  • 4.3. Preprocessing data
    • 4.3.1. Standardization, or mean removal and variance scaling
      • 4.3.1.1. Scaling features to a range
      • 4.3.1.2. Scaling sparse data
      • 4.3.1.3. Scaling data with outliers
      • 4.3.1.4. Centering kernel matrices
    • 4.3.2. Normalization
    • 4.3.3. Binarization
      • 4.3.3.1. Feature binarization
    • 4.3.4. Encoding categorical features
    • 4.3.5. Imputation of missing values
    • 4.3.6. Generating polynomial features
  • 4.4. Unsupervised dimensionality reduction
    • 4.4.1. PCA: principal component analysis
    • 4.4.2. Random projections
    • 4.4.3. Feature agglomeration
  • 4.5. Random Projection
    • 4.5.1. The Johnson-Lindenstrauss lemma
    • 4.5.2. Gaussian random projection
    • 4.5.3. Sparse random projection
  • 4.6. Kernel Approximation
    • 4.6.1. Nystroem Method for Kernel Approximation
    • 4.6.2. Radial Basis Function Kernel
    • 4.6.3. Additive Chi Squared Kernel
    • 4.6.4. Skewed Chi Squared Kernel
    • 4.6.5. Mathematical Details
  • 4.7. Pairwise metrics, Affinities and Kernels
    • 4.7.1. Cosine similarity
    • 4.7.2. Linear kernel
    • 4.7.3. Polynomial kernel
    • 4.7.4. Sigmoid kernel
    • 4.7.5. RBF kernel
    • 4.7.6. Chi-squared kernel
  • 4.8. Transforming the prediction target (y)
    • 4.8.1. Label binarization
    • 4.8.2. Label encoding
• 5. Dataset loading utilities
  • 5.1. General dataset API
  • 5.2. Toy datasets
  • 5.3. Sample images
  • 5.4. Sample generators
    • 5.4.1. Generators for classification and clustering
      • 5.4.1.1. Single label
      • 5.4.1.2. Multilabel
      • 5.4.1.3. Biclustering
    • 5.4.2. Generators for regression
    • 5.4.3. Generators for manifold learning
    • 5.4.4. Generators for decomposition
  • 5.5. Datasets in svmlight / libsvm format
  • 5.6. The Olivetti faces dataset
  • 5.7. The 20 newsgroups text dataset
    • 5.7.1. Usage
    • 5.7.2. Converting text to vectors
    • 5.7.3. Filtering text for more realistic training
  • 5.8. Downloading datasets from the mldata.org repository
  • 5.9. The Labeled Faces in the Wild face recognition dataset
    • 5.9.1. Usage
    • 5.9.2. Examples
  • 5.10. Forest covertypes
• 6. Strategies to scale computationally: bigger data
  • 6.1. Scaling with instances using out-of-core learning
    • 6.1.1. Streaming instances
    • 6.1.2. Extracting features
    • 6.1.3. Incremental learning
    • 6.1.4. Examples
    • 6.1.5. Notes
• 7. Computational Performance
  • 7.1. Prediction Latency
    • 7.1.1. Bulk versus Atomic mode
    • 7.1.2. Influence of the Number of Features
    • 7.1.3. Influence of the Input Data Representation
    • 7.1.4. Influence of the Model Complexity
    • 7.1.5. Feature Extraction Latency
  • 7.2. Prediction Throughput
  • 7.3. Tips and Tricks
    • 7.3.1. Linear algebra libraries
    • 7.3.2. Model Compression
    • 7.3.3. Model Reshaping
    • 7.3.4. Links