Scrapling Unifies HTTP, Browser, and Anti-Detection Crawling Under One Session Layer
Scrapling introduces a Session Dispatcher Pattern that decouples how data is fetched from how it is processed, letting a crawler switch from a fast HTTP call to a stealth browser at runtime without rewriting logic. The trade-off is a bus factor of one and deep coupling to a proxy-service ecosystem that funds the project.
A typical web scraper today drags in a tangle of dependencies: requests for HTTP, Playwright for JavaScript, Scrapy for scheduling, and custom scripts to dodge Cloudflare. Scrapling collapses that stack into one library by introducing a unified session layer. Three fetcher types—HTTP, dynamic browser, and stealth browser—share the same interface and return the same Response object, so a spider can route a request to the right channel with a single session ID parameter.
Under the hood, adaptive element tracking uses SQLite-stored fingerprints and difflib similarity matching, not machine learning. The Spider engine runs on anyio for async portability, checkpoints crawler state atomically to prevent corruption, and respects robots.txt Crawl-delay directives by taking the stricter of the user setting and the site requirement.
Benchmarks that claim a 700x speedup over BeautifulSoup are misleading; the real gap against Parsel is under 1%. The project is a solo effort with 1,450 commits from one developer, and its anti-detection strength depends heavily on paid proxy services, which also sponsor the project.
Scrapling’s real innovation is not any single fetcher but the Session Dispatcher Pattern: a runtime-routed abstraction that treats HTTP, browser, and stealth channels as interchangeable backends.
The adaptive element matching is deliberately low-tech—SQLite plus difflib—and that restraint makes it more trustworthy than a black-box ML approach would be.
Publishing benchmarks against BeautifulSoup when your parser is a Parsel derivative inflates the performance story; the honest comparison would be against Parsel, where the difference is negligible.
A solo-maintainer project with proxy-service sponsors faces a structural tension: the framework’s anti-detection value is tied to paid infrastructure the author has a commercial incentive to promote.
The anyio choice is a quiet signal of future-proofing; it costs little now but lets the engine run on trio if asyncio becomes a bottleneck or if the ecosystem shifts.