Even Two Outliers Can Distort Your Data Analysis
...And here's how to avoid drawing misleading conclusions.
In partnership with Sourcery
~92% of developers do code reviews manually. Moreover, only 8% of developers are fully confident in doing code reviews.
To make things even worse, ~80% of programmers wait more than one day for a review on their pull request.
Code review is a tedious job, and most developers are doing it the same way as 5 years ago.
Sourcery is making manual code reviews obsolete with AI.
Thus, every pull request instantly gets a human-like expert review from Sourcery with general feedback, in-line comments, and relevant suggestions.
What used to take over a day now takes a few seconds only.
Sourcery handles ~600,000+ requests every month (reviews + refactoring + writing tests + documenting code, etc.) and I have been using it for over 1.5 years now.
Thanks to Sourcery for partnering with me today!
Let’s get to today’s post now.
How outliers distort correlation and regression fit
Many data scientists solely rely on the correlation matrix to study the association between variables.
But unknown to them, the obtained statistic can be heavily driven by outliers.
This is evident from the image below.
Adding just two outliers drastically changed the correlation coefficient and the regression fit.
Thus, plotting the data is highly important.
This can save you from drawing wrong conclusions, which you may have drawn otherwise by solely looking at the summary statistics.
One thing that I often do when using a correlation matrix is creating a PairPlot as well (shown below).
This lets me infer if the scatter plot of two variables and their corresponding correlation measure resonate with each other or not.
We covered 8 more pitfalls and cautionary measures in this article: 8 Fatal (Yet Non-obvious) Pitfalls and Cautionary Measures in Data Science.
👉 Over to you: What is the difference between regression coefficient and correlation coefficient?
Are you overwhelmed with the amount of information in ML/DS?
Every week, I publish no-fluff deep dives on topics that truly matter to your skills for ML/DS roles.
For instance:
A Beginner-friendly Introduction to Kolmogorov Arnold Networks (KANs).
5 Must-Know Ways to Test ML Models in Production (Implementation Included).
Understanding LoRA-derived Techniques for Optimal LLM Fine-tuning
8 Fatal (Yet Non-obvious) Pitfalls and Cautionary Measures in Data Science
Implementing Parallelized CUDA Programs From Scratch Using CUDA Programming
You Are Probably Building Inconsistent Classification Models Without Even Realizing.
And many many more.
Join below to unlock all full articles:
SPONSOR US
Get your product in front of 79,000 data scientists and other tech professionals.
Our newsletter puts your products and services directly in front of an audience that matters — thousands of leaders, senior data scientists, machine learning engineers, data analysts, etc., who have influence over significant tech decisions and big purchases.
To ensure your product reaches this influential audience, reserve your space here or reply to this email to ensure your product reaches this influential audience.
potentially dumb question: is there a class techniques for loss functions that decrease the penalty of outliers? as opposed to robust techniques that just don't make the penalty as exteme. i realize the same could be done by removing outliers, and that this is only really a problem in gradient related optimization techniques, but could see it being helpful in high-dimension problems where outliers might not be as obvious so as to not update parameters so much cause of these outliers