55 lines
2.7 KiB
Markdown
55 lines
2.7 KiB
Markdown
- The average of guesses of a large number of people will typically be more accurate than an individual can be
|
|
- (wisdom of crowds)
|
|
- Bootstrapping
|
|
- generates simulated samples by sampling with replacement from existing sample
|
|
- sampling is random
|
|
- Out of bag sample
|
|
- data that was never picked in bootstrapping
|
|
- is often used for testing data (OOB samples were not in the training samples so the model has never seen them)
|
|
- with common techniques often ~30% of data is not picked (and forms OOB set)
|
|
- distribution of classes with bootstrapping will form a Gaussian distribution
|
|
- when enough samples are used bootstrap distributions will approximate population statistics
|
|
- Assumption is made with bootstrapping that the sample approximates the original population
|
|
- bootstrap samples are used as training sets, OOB samples serve as testing sets
|
|
```python
|
|
bootstrap_samples = [
|
|
resample(df, replace=True, n_samples=n random_state=i) for i in range(5)
|
|
]
|
|
```
|
|
- gets a list of dataframes with sampled data
|
|
- ensemble models can often greatly improve performance
|
|
- decision trees are often used as bases
|
|
- can increase computational complexity
|
|
- parallel ensembles
|
|
- base models are trained independently
|
|
- training is faster
|
|
- sequential ensembles
|
|
- models trained iteratively, adjusting for previous errors
|
|
- bagging
|
|
- (bootstrap aggregation)
|
|
- create bootstrap sets, train weak models and then aggregate predictions to get more accurate prediction
|
|
- decision trees in bagging have low bias but high variance
|
|
- bagging reduces model variance, not data variance
|
|
- (how much does model change if data changes slightly)
|
|
- bagging helps to make predictions more stable
|
|
- model variance should be low to increase prediction stability, data variance should be high so patterns are easier to find
|
|
- 10-25 decision trees are typically sufficient
|
|
- too many results in diminishing returns
|
|
- bagging gives high accuracy and better generalization but can be hard to interpret and have large computational costs
|
|
- random forests
|
|
- uses decision trees trained on feature subsets
|
|
- predictions are aggregated using averaging (regression) or majority vote (classification)
|
|
- features can also be selected at random as well as instances to decrease correlation between trees
|
|
- this can also increase computational efficiency
|
|
- prevents single features from dominating
|
|
- boosting
|
|
- base models fit iteratively, adjusting for previous error
|
|
- uses same training data without bootstrap sampling
|
|
- assigns higher weight to misclassified instances
|
|
- gradient boosting
|
|
- tries to minimize the gradient (derivative of the loss function) in each iteration
|
|
- learning rate determines model update strength (step sizes)
|
|
- ![[GradientDescent.excalidraw]]
|
|
|
|
## Zybooks notes
|
|
- |