initial commit

Running Start/CSB320 - Machine Learning Concepts/Class 4-23 (Model Validation & Evaluation).md

@@ -0,0 +1,60 @@
+#rs/notes #rs/class/csb320
+- - -
+- loss functions
+	- quantifies differences between predictions and observed values
+	- should minimize loss
+	- absolute loss
+		- prediction of instance
+	- log loss
+		- penalizes wrong predictions more harshly when they are more confident $$L_{\log}(y_{i}, \hat{p}_{i}) = -(y_{i}\ln(\hat{p}_{i}) + (1-y_{i})\ln(1-\hat{p}_{i}))$$
+	- cross-entropy loss
+		- used for multi-class classification
+		- measures difference between observed and predicted distributions
+	- type I error: false positive
+	- type II error: false negative
+- evaluation metrics
+	- accuracy $$\frac{\text{num correct predictions}}{\text{num incorrect predictions}}$$
+	- precision $$\frac{TP}{TP + FP}$$
+	- F-measures
+		- harmonic mean of precision and recall
+		- beta allows for emphasizing precision or recall in metric: $$F_{\beta} = (1 + \beta^2)\frac{{\text{precision} * \text{recall}}}{\beta^2 * \text{precision} + \text{recall}}$$
+		- $\beta>1$ emphasizes recall
+		- $\beta<1$ emphasizes precision
+	- kappa
+		- overall proportion of correct predictions
+		- evaluates performance when compared to random classifier
+		- 1: perfect classifier
+		- 0: same accuracy as random chance
+		- <0: worse than random chance
+	- accuracy fails for imbalanced datasets
+		- ex. when most people don't have cancer
+- AUC-ROC curve
+	- Receiver operating curve
+	- AUC represents the degree of seperability 
+	- ROC curves
+		- shows the trade off between true positive rate and false positive rate
+		- ![[rocCurve.excalidraw]]
+		- parametric graphs, false positive rate is on the x-axis and true positive rate is on the y-axis
+		- parameter is threshold for positive classification
+		- when threshold is raised there are less false positives but also less true positives
+			- vice versa
+		- area underneath the curve is a measure of the accuracy
+			- AUC (Area under the curve)
+- When to use metrics
+	- accuracy is very bad when dataset is not balanced
+	- precision vs. recall
+		- depends on the situation (don't want to have false negatives for cancer)
+	- F1 can maximize precision and recall and gives balance
+	- when balanced classes, maximize accuracy
+	- unbalanced classes can prioritize F1 on only one class if one is more important
+- Holdout method
+	- data is randomly partitioned into two independent sets
+	- validation set (often half of testing set) is used to decide optimal hyperparameter values
+- Random subsampling
+	- holdout is repeated k times and accuracy is average
+- Cross validation
+	- separate into sections, iterate through and use different section as the test set each time
+	- average the error
+	- if the average accuracy goes down with cross validation compared to holdout then model is likely overfitting
+- stratified cross-validaiton
+	- ensure that the distribution of data is the same in each section as in the general data set