Machine Learning

Pre-processing

Handling Missing Data: Filling missing values (e.g., using mean, median, mode, or interpolation).
Data Cleaning: Removing duplicates, fixing incorrect labels, correcting inconsistencies.
Scaling/Normalization: Standardizing or normalizing numerical features to ensure consistency.
Encoding Categorical Variables: Converting categorical data into numerical form (e.g., one-hot encoding, label encoding).
Handling Outliers: Removing or transforming extreme values that may distort the model.
Splitting Data: Dividing data into training, validation, and test sets.

Scaling

Use separate scalers for X and Y
- X and Y have different distributions (different scales and meanings)
- You can scale Y if it's a regression problem. Don't scale if it's a classification problem, since it's categorical
- Tree-based models like XGBoost, Decision Trees, or Random Forests usually don't need scaling because these models are not sensitive to feature scaling

Data Leakage

Divide training and test into separate datasets before performing scaling the features
- The mean and standard deviation used for scaling will be computed from the entire dataset.
- This means that information from the test set is indirectly influencing the training data.
- Your model will learn from statistics that it would not have access to in a real-world scenario.
- This can lead to overfitting and poor generalization.

Feature Engineering

PCA

Use PCA to reduce dimensionality
- Always scale the predictors before applying PCA
- PCA relies on the variance of the data to identify the principal components. If your predictors are on different scales, PCA may disproportionately weigh the features with larger scales
What's covariance matrix?
- A covariance matrix is a square matrix that contains the covariances between pairs of variables in a dataset.
- Covariance measures the degree to which two variables change together

Model Training

Model Selection

Which model is better? It depends on the problem at hand. If the relationship between the features and the response is well approximated by a linear model as in, then an approach such as linear regression will likely work well, and will outperform a method such as a regression tree that does not exploit this linear structure. If instead there is a highly non-linear and complex relationship between the features and the response as indicated by model, then decision trees may outperform classical approaches.

Model Performance

Prefer choosing models that have good cross-validation and test accuracy
- Good Cross-Validation Accuracy: a good cross-validation accuracy indicates good stability and generalization across different subsets of data
- Good Test Accuracy: the model generalizes well on unseen data
In classification models, the way to measure performance is based on accuracy, precision, recall (sensitivity), specificity, and f1 score
- Precision: Out of all the instances that the model predicted as positive, how many were actually positive?
  - Precision = TP / (TP + FP)
  - High Precision: Indicates that when the model predicts a positive class, it is often correct. This is crucial in applications where the cost of a false positive is high.
  - Low Precision: Suggests that the model frequently predicts positive incorrectly, leading to many false alarms.
- Recall (Sensitivity): Measures the proportion of actual positives that were correctly identified.
  - Recall = TP / (TP + FN)
- F1 Score: The harmonic mean of precision and recall, providing a balance between the two.
  - F1 Score = 2 x (Precision x Recall / (Precision + Recall))
- Importance in applications: In medical diagnosis, the diseases where a false positive can cause unnecessary stress or treatment, high precision is essential.

MSE

def mean_squared_error(y_true, y_pred):
    return np.mean((Y_test - prediction) ** 2)

R²

R² (coefficient of determination): measures how well your model explains the variance in the target variable

def r2_score(Y_true, Y_pred):
   residual_sum_of_squares = np.sum((Y_true - Y_pred) ** 2)
   total_sum_of_squares = np.sum((Y_true - np.mean(Y_true)) ** 2)
   return 1 - (residual_sum_of_squares / total_sum_of_squares)