Discriminative vs. Generative Models
A popular ML interview question.
FIRE-1 Web Agent to scrape the web like humans!
FIRE-1 Web Agent by Firecrawl allows you to scrape data while navigating complex websites, interacting with buttons, and even filling forms—just like a human would.
Completely hands-off!
You can also use the FIRE-1 Web Agent to extract structured data from the web. Firecrawl's /extract v2 endpoint does that.
Thanks to Firecrawl for partnering today!
Discriminative vs. Generative Models
Here’s a visual that depicts how generative and discriminative models differ:
We have seen this topic come up in several interviews, so today, let’s understand the details.
#1) Discriminative models
Discriminative models are primarily centered around discriminating between values of the outcome.
While their general notion appeals towards classification, regression models are also discriminative models.
For instance, even though linear regression does not involve a decision boundary between two classes, it is still discriminating between an outcome of “0.5” vs an outcome of “1.5” vs an outcome of “-0.6”.
Mathematically speaking, they maximize the conditional probability P(Y|X), which is read as follows: “Given an input X, maximize the probability of label Y.”
Popular examples include:
Logistic regression
Random Forest
Decision Trees, etc.
#2) Generative models
Generative models primarily focus on learning the class-conditional distribution.
Thus, they maximize the joint probability P(X, Y) by learning the class-conditional distribution P(X|Y):
Popular examples include:
Naive Bayes
Linear Discriminant Analysis (LDA)
Gaussian Mixture Models, etc.
We formulated Gaussian Mixture Models and implemented them from scratch here: Gaussian Mixture Models (GMMs).
Since generative models learn the underlying distribution, they can generate new samples.
For instance, consider a GAN:
Once the model has been trained, you can remove the discriminator and just use the generator to generate real-looking images:
However, this is not possible with discriminative models.
Furthermore, generative models possess discriminative properties, i.e., they can be used for classification tasks (if needed).
However, discriminative models do not possess generative properties.
Let’s consider an example to better understand them.
Imagine you are a language classification system.
There are two ways to classify languages:
Learn every language and then classify a new language based on acquired knowledge.
Understand some distinctive patterns in each language without truly learning the language. Once you do that, classify a new language based on the learned patterns.
The first approach is generative. This is because you learned the underlying distribution of each language.
In other words, you learned the joint distribution P(Words, Language).
Moreover, since you understand the underlying distribution, you can also generate new sentences.
The second approach is discriminative. This is because you only learned some distinctive patterns for each language:
If so and so words appear, it is likely “Langauge A.”
If this specific set of words appears, it is likely “Langauge B.”
and so on.
In this case, since you learned the conditional distribution P(Language|Words), you cannot generate new sentences.
This is the difference between generative and discriminative models.
Also, the above description might persuade you that generative models are more generally useful, but it is not true.
Generative models have their own modeling complications—like requiring much more data than discriminative models.
Relate it to the language classification example again.
Imagine the amount of data you would need to learn all languages (generative approach) vs. the amount of data you would need to understand some distinctive patterns (discriminative approach).
👉 Over to you: What are some other problems while training generative models?
Thanks for reading!
P.S. For those wanting to develop “Industry ML” expertise:
At the end of the day, all businesses care about impact. That’s it!
Can you reduce costs?
Drive revenue?
Can you scale ML models?
Predict trends before they happen?
We have discussed several other topics (with implementations) that align with such topics.
Here are some of them:
Learn how to build Agentic systems in an ongoing crash course with 10 parts.
Learn how to build real-world RAG apps and evaluate and scale them in this crash course.
Learn sophisticated graph architectures and how to train them on graph data in this crash course.
So many real-world NLP systems rely on pairwise context scoring. Learn scalable approaches here.
Learn how to run large models on small devices using Quantization techniques.
Learn how to generate prediction intervals or sets with strong statistical guarantees for increasing trust using Conformal Predictions.
Learn how to identify causal relationships and answer business questions using causal inference in this crash course.
Learn how to scale and implement ML model training in this practical guide.
Learn techniques to reliably test new models in production.
Learn how to build privacy-first ML systems using Federated Learning.
Learn 6 techniques with implementation to compress ML models.
All these resources will help you cultivate key skills that businesses and companies care about the most.