D4Vinci/Scrapling
Repository · https://github.com/d4vinci/scrapling
🕷️ The BEST Web Scraping framework that handles everything from a single request to a full-scale crawl! Don't be shy, join here: https://discord.gg/EMgGbDceNQ and follow here for daily tips and tricks: https://x.com/Scrapling_dev
First seen by Oriel on .
AI summary
Scrapling is a Python web-scraping framework combining fast HTTP fetching, stealth browser automation, parsing, concurrent crawling, exports, CLI access, and AI-agent integration for single pages or whole sites.
- Fetcher and StealthyFetcher support browser impersonation, JavaScript rendering, persistent sessions, proxy rotation, Cloudflare handling, DNS-over-HTTPS, remote CDP browsers, and background API-response capture.
- Its Scrapy-like Spider API provides configurable concurrency, per-domain AutoThrottle, blocked-request retries, robots.txt compliance, pause/resume checkpoints, streaming results, development caching, and multiple session types.
- Extraction supports CSS, XPath, text, regex, DOM navigation, adaptive element relocation, similar-element matching, generated selectors, and clean RAG-ready Markdown without an LLM.
- Built-in templates cover rules, sitemaps, XML/CSV feeds, Shopify stores, and website content, while results export to JSON, JSONL, CSV, or XML.
- The MCP server and Agent Skill let AI tools browse and extract data; the project reports 92% test coverage, full type hints, and 10x faster JSON serialization.
#python #web scraping #browser automation #llm agents
Generated by gpt-5.6-sol. Check the original for details.
Metric trends
Last 30 days · Up to 120 recorded samples per source and metric, with reposts kept separate · First and latest retained
Stars
Total repository stars at each observation.
- GitHub Trending: 98,504, latest observation ; +6,428 over observed period since
Stars gained (1d)
Daily star gain reported by GitHub Trending. A change compares two daily values; it is not additional stars between observations.
- GitHub Trending: 1,655, latest observation ; +763 over observed period since
Forks
Total repository forks at each observation.
- GitHub Trending: 10,312, latest observation ; +598 over observed period since
Aliases
| Source | External ID | URL |
|---|---|---|
| GitHub Trending | d4vinci/scrapling | https://github.com/D4Vinci/Scrapling Read discussion |