ben/ObsidianVault
1
0 Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

vault backup: 2026-05-20 11:36:54

Browse Source
This commit is contained in:
ben committed 2026-05-20 11:36:54 -07:00
1 parent 237dc8e19f
commit 6c22b3d556
1 file changed
+66 -58
Download Patch File Download Diff File Expand all files Collapse all files
Running Start/CSB320 - Machine Learning Concepts/Class 4-23 (Model Validation & Evaluation).md
+66 -58
View File
@@ -1,60 +1,68 @@
#rs/notes #rs/class/csb320 #rs/notes #rs/class/csb320
- - - - - -
- loss functions
- quantifies differences between predictions and observed values > [!NOTE]- Bullet Notes
- should minimize loss > - loss functions
- absolute loss > - quantifies differences between predictions and observed values
- prediction of instance > - should minimize loss
- log loss > - absolute loss
- penalizes wrong predictions more harshly when they are more confident $$L_{\log}(y_{i}, \hat{p}_{i}) = -(y_{i}\ln(\hat{p}_{i}) + (1-y_{i})\ln(1-\hat{p}_{i}))$$ > - prediction of instance
- cross-entropy loss > - log loss
- used for multi-class classification > - penalizes wrong predictions more harshly when they are more confident $$L_{\log}(y_{i}, \hat{p}_{i}) = -(y_{i}\ln(\hat{p}_{i}) + (1-y_{i})\ln(1-\hat{p}_{i}))$$
- measures difference between observed and predicted distributions > - cross-entropy loss
- type I error: false positive > - used for multi-class classification
- type II error: false negative > - measures difference between observed and predicted distributions
- evaluation metrics > - type I error: false positive
- accuracy $$\frac{\text{num correct predictions}}{\text{num incorrect predictions}}$$ > - type II error: false negative
- precision $$\frac{TP}{TP + FP}$$ > - evaluation metrics
- F-measures > - accuracy $$\frac{\text{num correct predictions}}{\text{num incorrect predictions}}$$
- harmonic mean of precision and recall > - precision $$\frac{TP}{TP + FP}$$
- beta allows for emphasizing precision or recall in metric: $$F_{\beta} = (1 + \beta^2)\frac{{\text{precision} * \text{recall}}}{\beta^2 * \text{precision} + \text{recall}}$$ > - F-measures
- $\beta>1$ emphasizes recall > - harmonic mean of precision and recall
- $\beta<1$ emphasizes precision > - beta allows for emphasizing precision or recall in metric: $$F_{\beta} = (1 + \beta^2)\frac{{\text{precision} * \text{recall}}}{\beta^2 * \text{precision} + \text{recall}}$$
- kappa > - $\beta>1$ emphasizes recall
- overall proportion of correct predictions > - $\beta<1$ emphasizes precision
- evaluates performance when compared to random classifier > - kappa
- 1: perfect classifier > - overall proportion of correct predictions
- 0: same accuracy as random chance > - evaluates performance when compared to random classifier
- <0: worse than random chance > - 1: perfect classifier
- accuracy fails for imbalanced datasets > - 0: same accuracy as random chance
- ex. when most people don't have cancer > - <0: worse than random chance
- AUC-ROC curve > - accuracy fails for imbalanced datasets
- Receiver operating curve > - ex. when most people don't have cancer
- AUC represents the degree of seperability > - AUC-ROC curve
- ROC curves > - Receiver operating curve
- shows the trade off between true positive rate and false positive rate > - AUC represents the degree of seperability
- ![[rocCurve.excalidraw]] > - ROC curves
- parametric graphs, false positive rate is on the x-axis and true positive rate is on the y-axis > - shows the trade off between true positive rate and false positive rate
- parameter is threshold for positive classification > - ![[rocCurve.excalidraw]]
- when threshold is raised there are less false positives but also less true positives > - parametric graphs, false positive rate is on the x-axis and true positive rate is on the y-axis
- vice versa > - parameter is threshold for positive classification
- area underneath the curve is a measure of the accuracy > - when threshold is raised there are less false positives but also less true positives
- AUC (Area under the curve) > - vice versa
- When to use metrics > - area underneath the curve is a measure of the accuracy
- accuracy is very bad when dataset is not balanced > - AUC (Area under the curve)
- precision vs. recall > - When to use metrics
- depends on the situation (don't want to have false negatives for cancer) > - accuracy is very bad when dataset is not balanced
- F1 can maximize precision and recall and gives balance > - precision vs. recall
- when balanced classes, maximize accuracy > - depends on the situation (don't want to have false negatives for cancer)
- unbalanced classes can prioritize F1 on only one class if one is more important > - F1 can maximize precision and recall and gives balance
- Holdout method > - when balanced classes, maximize accuracy
- data is randomly partitioned into two independent sets > - unbalanced classes can prioritize F1 on only one class if one is more important
- validation set (often half of testing set) is used to decide optimal hyperparameter values > - Holdout method
- Random subsampling > - data is randomly partitioned into two independent sets
- holdout is repeated k times and accuracy is average > - validation set (often half of testing set) is used to decide optimal hyperparameter values
- Cross validation > - Random subsampling
- separate into sections, iterate through and use different section as the test set each time > - holdout is repeated k times and accuracy is average
- average the error > - Cross validation
- if the average accuracy goes down with cross validation compared to holdout then model is likely overfitting > - separate into sections, iterate through and use different section as the test set each time
- stratified cross-validaiton > - average the error
- ensure that the distribution of data is the same in each section as in the general data set > - if the average accuracy goes down with cross validation compared to holdout then model is likely overfitting
> - stratified cross-validaiton
> - ensure that the distribution of data is the same in each section as in the general data set
# Model Validation and Evaluation
There are many ways to evaluate the performance of models. This is necessary to ensure that they are performing at an acceptable level and to help to tune them with different hyperparameters, preprocessing, data, and other pipeline changes.
## Loss Functions
Loss functions are meant to quantify the difference between observed values and the predictions of a model. They can be used to minimize loss and improve model performance.
### Absolute Loss