ObsidianVault/Running Start/CSB320 - Machine Learning Concepts/Class 5-21 (Ensemble Models).md

#rs/notes #rs/class/csb320

---

• The average of guesses of a large number of people will typically be more accurate than an individual can be
  • (wisdom of crowds)
• Bootstrapping
  • generates simulated samples by sampling with replacement from existing sample
  • sampling is random
  • Out of bag sample
    • data that was never picked in bootstrapping
    • is often used for testing data (OOB samples were not in the training samples so the model has never seen them)
    • with common techniques often ~30% of data is not picked (and forms OOB set)
  • distribution of classes with bootstrapping will form a Gaussian distribution
  • when enough samples are used bootstrap distributions will approximate population statistics
    • Assumption is made with bootstrapping that the sample approximates the original population
  • bootstrap samples are used as training sets, OOB samples serve as testing sets

bootstrap_samples = [
	resample(df, replace=True, n_samples=n random_state=i) for i in range(5)
]

• gets a list of dataframes with sampled data
• ensemble models can often greatly improve performance
  • decision trees are often used as bases
  • can increase computational complexity
• parallel ensembles
  • base models are trained independently
  • training is faster
• sequential ensembles
  • models trained iteratively, adjusting for previous errors
• bagging
  • (bootstrap aggregation)
  • create bootstrap sets, train weak models and then aggregate predictions to get more accurate prediction
  • decision trees in bagging have low bias but high variance
  • bagging reduces model variance, not data variance
    • (how much does model change if data changes slightly)
  • bagging helps to make predictions more stable
  • model variance should be low to increase prediction stability, data variance should be high so patterns are easier to find
  • 10-25 decision trees are typically sufficient
    • too many results in diminishing returns
  • bagging gives high accuracy and better generalization but can be hard to interpret and have large computational costs
• random forests
  • uses decision trees trained on feature subsets
  • predictions are aggregated using averaging (regression) or majority vote (classification)
  • features can also be selected at random as well as instances to decrease correlation between trees
    • this can also increase computational efficiency
    • prevents single features from dominating
• boosting
  • base models fit iteratively, adjusting for previous error
  • uses same training data without bootstrap sampling
  • assigns higher weight to misclassified instances
• gradient boosting
  • tries to minimize the gradient (derivative of the loss function) in each iteration
  • learning rate determines model update strength (step sizes)
  • !GradientDescent.excalidraw

Zybooks notes

• Boosting
  • decision stumps are used (decision trees with 1 layer)
  • an initial model is fit to the data, it will get many things wrong since it is a weak learner
  • Then, incorrectly predicted instances are weighted higher than correct ones and a new model is trained with this
  • This keeps happening and final predictions are made by going back and doing a weighted aggregation of all previous models to create a complex decision boundary from the decision stumps