A Single Frame Summary of 10 Most Common Regression and Classification Loss Functions

10 must know loss functions, along with their limitations and benefits.

Dec 22, 2023

Loss functions are a key component of ML algorithms.

They specify the objective an algorithm should aim to optimize during its training. In other words, loss functions tell the algorithm what it should minimize or maximize to improve its performance.

Model training with an objective function

Therefore, being aware of the most common loss functions is extremely crucial.

The visual below depicts the most commonly used loss functions for regression and classification tasks.

Regression

Mean Bias Error
1. Captures the average bias in the prediction.
2. However, it is rarely used in training ML models.
3. This is because negative errors may cancel positive errors, leading to zero loss, and consequently, no weight updates.
4. Mean bias error is foundational to the more advanced regression losses discussed below.
Mean Absolute Error (or L1 loss)
1. Measures the average absolute difference between predicted and actual value.
2. Positive errors and negative errors don’t cancel out.
3. One caveat is that small errors are as important as big ones. Thus, the magnitude of the gradient is independent of error size.
Mean Squared Error (or L2 loss)
1. It measures the squared difference between predicted and actual value.
2. Larger errors contribute more significantly than smaller errors.
3. The above point may also be a caveat as it is sensitive to outliers.
4. Yet, it is among the most common loss functions for many regression models. If you want to understand the origin of mean squared error, we discussed it in this newsletter issue: Why Mean Squared Error (MSE)?
Root Mean Squared Error
1. Mean Squared Error with a square root.
2. Loss and the dependent variable (y) have the same units.
Huber Loss
1. It is a combination of mean absolute error and mean squared error.
2. For smaller errors, mean squared error is used, which is differentiable through (unlike MAE, which is non-differentiable at x=0).
3. For large errors, mean absolute error is used, which is less sensitive to outliers.
4. One caveat is that it is parameterized — adding another hyperparameter to the list.
5. Read this full issue to learn more about Huber Regression: A Simple Technique to Robustify Linear Regression to Outliers
Log Cosh Loss
1. For small errors, log cash loss is approximately → x²/2 — quadratic.
2. For large errors, log cash loss is approximately → |x| - log(2) — linear.
3. Thus, it is very similar to Huber loss.
4. Also, it is non-parametric.
5. The only caveat is that it is a bit computationally expensive.

Classification

Binary cross entropy (BCE) or Log loss
1. A loss function used for binary classification tasks.
2. Measures the dissimilarity between predicted probabilities and true binary labels, through the logarithmic loss.
3. Where did the log loss originate from? We discussed it here: why Do We Use log-loss To Train Logistic Regression?
Hinge Loss
1. Penalizes both wrong and right (but less confident) predictions).
2. It is based on the concept of margin, which represents the distance between a data point and the decision boundary.
3. The larger the margin, the more confident the classifier is about its prediction.
4. Particularly used to train support vector machines (SVMs).
Cross-Entropy Loss
1. An extension of Binary Cross Entropy loss to multi-class classification tasks.
KL Divergence
1. Measure information lost when one distribution is approximated using another distribution.
2. The more information is lost, the more the KL Divergence.
3. For classification, however, using KL divergence is the same as minimizing cross entropy (proved below):
4. Thus, it is recommended to use cross-entropy loss directly.
5. That said, KL divergence is widely used in many other algorithms:
  1. As a loss function in the t-SNE algorithm. We discussed it here: t-SNE article.
  2. For model compression using knowledge distillation. We discussed it here: Model compression article.

A couple of months back, I also published an algorithm-wise loss function summary, which you can read in this issue: An Algorithm-wise Summary of Loss Functions in Machine Learning.

👉 Over to you: What other common loss functions have I missed?

👉 If you liked this post, don’t forget to leave a like ❤️. It helps more people discover this newsletter on Substack and tells me that you appreciate reading these daily insights.