Advanced Evaluation Metrics

What are Evaluation Metrics?

An evaluation metric is a quantitative measure that tells you how well your model's predictions match reality. Choosing the right metric is as important as choosing the right algorithm — a model can score 99% accuracy while being completely useless if the dataset is imbalanced, or it can minimize MSE while producing predictions that are practically wrong.

Metrics are divided by problem type: regression metrics measure error in continuous predictions, while classification metrics measure correctness and trade-offs in categorical predictions.

Regression Metrics (Continuous Predictions)

Mean Squared Error (MSE)

MSE = (1/n) * Σ(actual - predicted)²

• Pros: Mathematical convenience, differentiable (great for Gradient Descent).
• Cons: Penalizes large errors heavily (outliers have huge impact).
• Use when: You want to punish large mistakes harshly.

Root Mean Squared Error (RMSE)

RMSE = √MSE = √((1/n) * Σ(actual - predicted)²)

• Pros: Same units as target variable (interpretable). If target is "Price in dollars", RMSE is also in dollars.
• Cons: Still sensitive to outliers.
• Use when: You need interpretable errors in original units.

Mean Absolute Error (MAE)

MAE = (1/n) * Σ|actual - predicted|

• Pros: Robust to outliers. Each error contributes equally.
• Cons: Not differentiable at 0 (harder to optimize).
• Use when: You want robustness and interpretability, or data has outliers.

Classification Metrics (Categorical Predictions)

AUC-ROC Curve

ROC = Receiver Operating Characteristic.

• X-axis: False Positive Rate (FPR) = FP / (FP + TN)
• Y-axis: True Positive Rate (TPR) = TP / (TP + FN)
• AUC (Area Under Curve):
  • 1.0 = Perfect classifier
  • 0.5 = Random guessing
  • <0.5 = Worse than random (flip predictions!)

Use ROC-AUC when classes are slightly imbalanced.

Precision-Recall Curve

• X-axis: Recall = TP / (TP + FN)
• Y-axis: Precision = TP / (TP + FP)
• PR-AUC: Area under Precision-Recall curve.

Use PR-AUC when classes are heavily imbalanced (e.g., fraud detection: 0.01% fraud vs 99.99% normal).