ObsidianVault/Running Start/CSB320 - Machine Learning Concepts/Class 4-23 (Model Validation & Evaluation).md

#rs/notes #rs/class/csb320

---

• loss functions
  • quantifies differences between predictions and observed values
  • should minimize loss
  • absolute loss
    • prediction of instance
  • log loss
    • penalizes wrong predictions more harshly when they are more confident L_{\log}(y_{i}, \hat{p}_{i}) = -(y_{i}\ln(\hat{p}_{i}) + (1-y_{i})\ln(1-\hat{p}_{i}))
  • cross-entropy loss
    • used for multi-class classification
    • measures difference between observed and predicted distributions
  • type I error: false positive
  • type II error: false negative
• evaluation metrics
  • accuracy \frac{\text{num correct predictions}}{\text{num incorrect predictions}}
  • precision \frac{TP}{TP + FP}
  • F-measures
    • harmonic mean of precision and recall
    • beta allows for emphasizing precision or recall in metric: F_{\beta} = (1 + \beta^2)\frac{{\text{precision} * \text{recall}}}{\beta^2 * \text{precision} + \text{recall}}
    • \beta>1 emphasizes recall
    • \beta<1 emphasizes precision
  • kappa
    • overall proportion of correct predictions
    • evaluates performance when compared to random classifier
    • 1: perfect classifier
    • 0: same accuracy as random chance
    • <0: worse than random chance
  • accuracy fails for imbalanced datasets
    • ex. when most people don't have cancer
• AUC-ROC curve
  • Receiver operating curve
  • AUC represents the degree of seperability
  • ROC curves
    • shows the trade off between true positive rate and false positive rate
    • !rocCurve.excalidraw
    • parametric graphs, false positive rate is on the x-axis and true positive rate is on the y-axis
    • parameter is threshold for positive classification
    • when threshold is raised there are less false positives but also less true positives
      • vice versa
    • area underneath the curve is a measure of the accuracy
      • AUC (Area under the curve)
• When to use metrics
  • accuracy is very bad when dataset is not balanced
  • precision vs. recall
    • depends on the situation (don't want to have false negatives for cancer)
  • F1 can maximize precision and recall and gives balance
  • when balanced classes, maximize accuracy
  • unbalanced classes can prioritize F1 on only one class if one is more important
• Holdout method
  • data is randomly partitioned into two independent sets
  • validation set (often half of testing set) is used to decide optimal hyperparameter values
• Random subsampling
  • holdout is repeated k times and accuracy is average
• Cross validation
  • separate into sections, iterate through and use different section as the test set each time
  • average the error
  • if the average accuracy goes down with cross validation compared to holdout then model is likely overfitting
• stratified cross-validaiton
  • ensure that the distribution of data is the same in each section as in the general data set