Open-source Python Development Landscape

30 must-know tools for Python development.

Web scraping in pure English with Firecrawl Extract

Using the Firecrawl Extract endpoint, just describe what you want to extract in a prompt. This produces LLM-ready structured output. No more hard coding!

FireCrawl also generates code snippets to run the same job programmatically.

Thanks to FireCrawl for showing us their powerful scraping capabilities and partnering today!

---

Open-source Python Development Landscape

Python development involves various stages and equally many tools to manage them:

• For dependencies, tools like pip, Conda, and Poetry help.

• For profiling, psutil and Scalene have been popular. But the newly launched Perforator is providing a much more reliable production-ready continuous profiling app to collect CPU profiles without affecting performance.

• And more.

We gathered several such popular tools in the following visual:

The objective is to provide a structured overview of the essential tools developers can use across different aspects of Python development.

Let's explore each category and its top tools in more detail.

1. Dependency & Package Managers

Manage Python package installations and dependencies.

• pip – The default package manager for Python, used to install and manage libraries.

• Conda – A package, dependency, and environment manager supporting multiple programming languages.

• Poetry – A dependency management tool that simplifies package management and publishing.

2. Monitoring and Profiling

Optimize and analyze performance.

• psutil – Provides system and process utilities for monitoring resource usage.

• Perforator – A cluster-wide continuous profiling tool designed for large data centers.

• Scalene – A high-performance Python profiler for CPU, memory, and GPU usage.

3. Virtual Environments

Ensure project isolation and manage dependencies efficiently.

• pyenv – Helps manage multiple Python versions on a system.

• virtualenv – Creates isolated Python environments to manage dependencies efficiently.

• pipenv – Combines pip and virtualenv for better dependency management.

4. Linters & Style Checkers

Enforce coding standards and maintain code quality.

• Pylint – Checks for errors and enforces coding standards.

• Ruff – A fast Python linter and formatter.

• Flake8 – Combines various linting tools into a single framework.

5. Type Checkers

Ensure type correctness in Python codebases.

• mypy – A static type checker for Python.

• Pyright – A fast type checker developed by Microsoft.

• Typeguard – A runtime type checker for Python.

6. Logging

Monitor application behavior and track issues.

• Rich – Provides beautiful console output with formatting, tables, and progress bars.

• Loguru – A modern logging framework for Python.

• tqdm – A simple, fast progress bar for loops.

7. Testing

Automate testing for software reliability.

• pytest – A widely used testing framework for Python.

• hypothesis – A property-based testing library.

• Robot Framework – A keyword-driven testing framework.

8. Debugging

Identify and fix issues in your code.

• PDB – The built-in Python debugger.

• Icecream – A debugging tool that prints variables and expressions.

• PySnooper – A lightweight debugging utility that logs function execution details.

9. Code Refactoring

Improve and restructure code efficiently.

• Jedi – An autocompletion and static analysis library for Python.

• Sourcery – An AI-powered code refactoring tool.

• Vulture – Finds unused code in Python projects.

10. Code Security

Detect and mitigate security vulnerabilities.

• Bandit – A security linter for identifying vulnerabilities in Python code.

• Safety – Checks Python dependencies for security vulnerabilities.

• Detect-Secrets – Scans code for hardcoded secrets.

These tools are invaluable for any Python developer, helping with everything from virtual environments and dependency management to debugging, logging, and security.

Incorporating them into your workflow can significantly improve your development experience and code quality.

👉 Over to you: Which tools do you regularly use from this landscape?

Thanks for reading!

---

P.S. For those wanting to develop “Industry ML” expertise:

At the end of the day, all businesses care about impact. That’s it!

• Can you reduce costs?

• Drive revenue?

• Can you scale ML models?

• Predict trends before they happen?

We have discussed several other topics (with implementations) in the past that align with such topics.

Here are some of them:

• Learn sophisticated graph architectures and how to train them on graph data: A Crash Course on Graph Neural Networks – Part 1.

• So many real-world NLP systems rely on pairwise context scoring. Learn scalable approaches here: Bi-encoders and Cross-encoders for Sentence Pair Similarity Scoring – Part 1.

• Learn techniques to run large models on small devices: Quantization: Optimize ML Models to Run Them on Tiny Hardware.

• Learn how to generate prediction intervals or sets with strong statistical guarantees for increasing trust: Conformal Predictions: Build Confidence in Your ML Model’s Predictions.

• Learn how to identify causal relationships and answer business questions: A Crash Course on Causality – Part 1

• Learn how to scale ML model training: A Practical Guide to Scaling ML Model Training.

• Learn techniques to reliably roll out new models in production: 5 Must-Know Ways to Test ML Models in Production (Implementation Included)

• Learn how to build privacy-first ML systems: Federated Learning: A Critical Step Towards Privacy-Preserving Machine Learning.

• Learn how to compress ML models and reduce costs: Model Compression: A Critical Step Towards Efficient Machine Learning.

All these resources will help you cultivate key skills that businesses and companies care about the most.