Oriel — AI & technology worth noticing, every day.

D4Vinci/Scrapling

Repository · https://github.com/d4vinci/scrapling

🕷️ The BEST Web Scraping framework that handles everything from a single request to a full-scale crawl! Don't be shy, join here: https://discord.gg/EMgGbDceNQ and follow here for daily tips and tricks: https://x.com/Scrapling_dev

First seen by Oriel on .

AI summary

Scrapling is a Python web-scraping framework combining fast HTTP fetching, stealth browser automation, parsing, concurrent crawling, exports, CLI access, and AI-agent integration for single pages or whole sites.

  • Fetcher and StealthyFetcher support browser impersonation, JavaScript rendering, persistent sessions, proxy rotation, Cloudflare handling, DNS-over-HTTPS, remote CDP browsers, and background API-response capture.
  • Its Scrapy-like Spider API provides configurable concurrency, per-domain AutoThrottle, blocked-request retries, robots.txt compliance, pause/resume checkpoints, streaming results, development caching, and multiple session types.
  • Extraction supports CSS, XPath, text, regex, DOM navigation, adaptive element relocation, similar-element matching, generated selectors, and clean RAG-ready Markdown without an LLM.
  • Built-in templates cover rules, sitemaps, XML/CSV feeds, Shopify stores, and website content, while results export to JSON, JSONL, CSV, or XML.
  • The MCP server and Agent Skill let AI tools browse and extract data; the project reports 92% test coverage, full type hints, and 10x faster JSON serialization.

#python #web scraping #browser automation #llm agents

Generated by gpt-5.6-sol. Check the original for details.

Metric trends

Last 30 days · Up to 120 recorded samples per source and metric, with reposts kept separate · First and latest retained

Stars

Total repository stars at each observation.

  • GitHub Trending: 98,504, latest observation ; +6,428 over observed period since

Stars gained (1d)

Daily star gain reported by GitHub Trending. A change compares two daily values; it is not additional stars between observations.

  • GitHub Trending: 1,655, latest observation ; +763 over observed period since

Forks

Total repository forks at each observation.

  • GitHub Trending: 10,312, latest observation ; +598 over observed period since

Aliases

SourceExternal IDURL
GitHub Trendingd4vinci/scraplinghttps://github.com/D4Vinci/Scrapling Read discussion