🔥Generate LLMs.txt from any website with FireCrawl
FireCrawl has released the /llmstxt
endpoint that allows you to transform any website into LLM-ready text files.
Simply provide a URL, and Firecrawl will crawl the site and generate both llms.txt
and llms-full.txt
files that can be used for training or analysis with any LLM.
Here’s a sample usage where I scraped this newsletter:
Try the LLM.txt Generator API here →
Thanks to FireCrawl for partnering with us today!
12 Powerful Tools For AI Agents
AI Agents use LLM as their brains and tools like their hands to take action.
When it comes to building Agents, CrewAI is one of my favorite frameworks.
Our entire AI Agent crash course is also based on CrewAI.
Here are 12 built-in tools that come with CrewAI:
#1) File Read Tool → Reads and extracts structured or unstructured data from files, making it accessible for AI agents.
#2) File Writer Tool → Writes data or content to files, enabling AI agents to store information dynamically.
#3) Code Interpreter Tool → Executes Python code generated by an AI agent, allowing dynamic computations and scripting.
#4) Scrape Website Tool → Extracts and processes content from web pages, enabling AI-powered web data retrieval.
#5) Serper Dev Tool → Fetches real-time search results from the internet, enhancing AI's access to up-to-date information.
#6) Directory Read Tool → Lists and explores the contents of a directory, helping AI agents navigate file systems.
#7) Firecrawl Search Tool → Searches websites and converts them into Markdown format for easy processing and storage.
#8) Browserbase Load Tool → Runs, manages, and monitors headless browsers, allowing AI to interact with web pages dynamically.
#9) PDF Search Tool → Enables RAG-based search within PDFs, making document retrieval more efficient.
#10) GitHub Search Tool → Performs RAG-powered searches in GitHub repositories to find relevant code and documentation.
#11) TXT Search Tool → Searches text files efficiently using RAG techniques to extract relevant information.
#12) NL2SQL Tool → Converts natural language queries into SQL commands, making database interactions more intuitive.
Just like the RAG crash course, we have been covering everything about AI agents in detail to fully equip you with building agentic systems:
Fundamentals (Why Agents? The Motivation; Create Agents, Tasks, Tools, etc.)
Memory for Agents
Agentic Flows
Guardrails for Agents
Implementing Agentic design patterns.
Agentic RAG
Optimizing Agents for production, and more.
In this full crash course, we shall cover everything you need to know about building robust Agentic systems, starting from the fundamentals. We cover everything in a practical and beginner-friendly way.
👉 Over to you: What other powerful tools have we missed?
Thanks for reading!
P.S. For those wanting to develop “Industry ML” expertise:
At the end of the day, all businesses care about impact. That’s it!
Can you reduce costs?
Drive revenue?
Can you scale ML models?
Predict trends before they happen?
We have discussed several other topics (with implementations) that align with such topics.
Here are some of them:
Learn sophisticated graph architectures and how to train them on graph data: A Crash Course on Graph Neural Networks – Part 1.
So many real-world NLP systems rely on pairwise context scoring. Learn scalable approaches here: Bi-encoders and Cross-encoders for Sentence Pair Similarity Scoring – Part 1.
Learn techniques to run large models on small devices: Quantization: Optimize ML Models to Run Them on Tiny Hardware.
Learn how to generate prediction intervals or sets with strong statistical guarantees for increasing trust: Conformal Predictions: Build Confidence in Your ML Model’s Predictions.
Learn how to identify causal relationships and answer business questions: A Crash Course on Causality – Part 1
Learn how to scale ML model training: A Practical Guide to Scaling ML Model Training.
Learn techniques to reliably roll out new models in production: 5 Must-Know Ways to Test ML Models in Production (Implementation Included)
Learn how to build privacy-first ML systems: Federated Learning: A Critical Step Towards Privacy-Preserving Machine Learning.
Learn how to compress ML models and reduce costs: Model Compression: A Critical Step Towards Efficient Machine Learning.
All these resources will help you cultivate key skills that businesses and companies care about the most.