3. Supervised learning

• 3.1. Generalized Linear Models
  • 3.1.1. Ordinary Least Squares
    • 3.1.1.1. Ordinary Least Squares Complexity
  • 3.1.2. Ridge Regression
    • 3.1.2.1. Ridge Complexity
    • 3.1.2.2. Setting the regularization parameter: generalized Cross-Validation
  • 3.1.3. Lasso
    • 3.1.3.1. Setting regularization parameter
      • 3.1.3.1.1. Using cross-validation
      • 3.1.3.1.2. Information-criteria based model selection
  • 3.1.4. Elastic Net
  • 3.1.5. Multi-task Lasso
  • 3.1.6. Least Angle Regression
  • 3.1.7. LARS Lasso
    • 3.1.7.1. Mathematical formulation
  • 3.1.8. Orthogonal Matching Pursuit (OMP)
  • 3.1.9. Bayesian Regression
    • 3.1.9.1. Bayesian Ridge Regression
    • 3.1.9.2. Automatic Relevance Determination - ARD
  • 3.1.10. Logistic regression
  • 3.1.11. Stochastic Gradient Descent - SGD
  • 3.1.12. Perceptron
  • 3.1.13. Passive Aggressive Algorithms
• 3.2. Support Vector Machines
  • 3.2.1. Classification
    • 3.2.1.1. Multi-class classification
    • 3.2.1.2. Unbalanced problems
  • 3.2.2. Regression
  • 3.2.3. Density estimation, novelty detection
  • 3.2.4. Complexity
  • 3.2.5. Tips on Practical Use
  • 3.2.6. Kernel functions
    • 3.2.6.1. Custom Kernels
      • 3.2.6.1.1. Using Python functions as kernels
      • 3.2.6.1.2. Using the Gram matrix
      • 3.2.6.1.3. Parameters of the RBF Kernel
  • 3.2.7. Mathematical formulation
    • 3.2.7.1. SVC
    • 3.2.7.2. NuSVC
  • 3.2.8. Implementation details
• 3.3. Stochastic Gradient Descent
  • 3.3.1. Classification
  • 3.3.2. Regression
  • 3.3.3. Stochastic Gradient Descent for sparse data
  • 3.3.4. Complexity
  • 3.3.5. Tips on Practical Use
  • 3.3.6. Mathematical formulation
    • 3.3.6.1. SGD
  • 3.3.7. Implementation details
• 3.4. Nearest Neighbors
  • 3.4.1. Unsupervised Nearest Neighbors
  • 3.4.2. Nearest Neighbors Classification
  • 3.4.3. Nearest Neighbors Regression
  • 3.4.4. Nearest Neighbor Algorithms
    • 3.4.4.1. Brute Force
    • 3.4.4.2. K-D Tree
    • 3.4.4.3. Ball Tree
    • 3.4.4.4. Choice of Nearest Neighbors Algorithm
    • 3.4.4.5. Effect of leaf_size
  • 3.4.5. Nearest Centroid Classifier
    • 3.4.5.1. Nearest Shrunken Centroid
• 3.5. Gaussian Processes
  • 3.5.1. Examples
    • 3.5.1.1. An introductory regression example
    • 3.5.1.2. Fitting Noisy Data
  • 3.5.2. Mathematical formulation
    • 3.5.2.1. The initial assumption
    • 3.5.2.2. The best linear unbiased prediction (BLUP)
    • 3.5.2.3. The empirical best linear unbiased predictor (EBLUP)
  • 3.5.3. Correlation Models
  • 3.5.4. Regression Models
  • 3.5.5. Implementation details
• 3.6. Partial Least Squares
• 3.7. Naive Bayes
  • 3.7.1. Gaussian Naive Bayes
  • 3.7.2. Multinomial Naive Bayes
  • 3.7.3. Bernoulli Naive Bayes
• 3.8. Decision Trees
  • 3.8.1. Classification
  • 3.8.2. Regression
  • 3.8.3. Multi-output problems
  • 3.8.4. Complexity
  • 3.8.5. Tips on practical use
  • 3.8.6. Tree algorithms: ID3, C4.5, C5.0 and CART
  • 3.8.7. Mathematical formulation
    • 3.8.7.1. Classification criteria
    • 3.8.7.2. Regression criteria
• 3.9. Ensemble methods
  • 3.9.1. Forests of randomized trees
    • 3.9.1.1. Random Forests
    • 3.9.1.2. Extremely Randomized Trees
    • 3.9.1.3. Parameters
    • 3.9.1.4. Parallelization
    • 3.9.1.5. Feature importance evaluation
    • 3.9.1.6. Totally Random Trees Embedding
  • 3.9.2. Gradient Tree Boosting
    • 3.9.2.1. Classification
    • 3.9.2.2. Regression
    • 3.9.2.3. Mathematical formulation
      • 3.9.2.3.1. Loss Functions
    • 3.9.2.4. Regularization
      • 3.9.2.4.1. Shrinkage
      • 3.9.2.4.2. Subsampling
    • 3.9.2.5. Interpretation
      • 3.9.2.5.1. Feature importance
      • 3.9.2.5.2. Partial dependence
• 3.10. Multiclass and multilabel algorithms
  • 3.10.1. One-Vs-The-Rest
    • 3.10.1.1. Multilabel learning with OvR
  • 3.10.2. One-Vs-One
  • 3.10.3. Error-Correcting Output-Codes
• 3.11. Feature selection
  • 3.11.1. Univariate feature selection
  • 3.11.2. Recursive feature elimination
  • 3.11.3. L1-based feature selection
    • 3.11.3.1. Selecting non-zero coefficients
    • 3.11.3.2. Randomized sparse models
  • 3.11.4. Tree-based feature selection
• 3.12. Semi-Supervised
  • 3.12.1. Label Propagation
• 3.13. Linear and Quadratic Discriminant Analysis
  • 3.13.1. Dimensionality Reduction using LDA
  • 3.13.2. Mathematical Idea
• 3.14. Isotonic regression