跪拜 Guibai
← Back to the summary

Scrapling Unifies HTTP, Browser, and Anti-Detection Crawling Under One Session Layer

Hello everyone, I'm Ruofeng.

In the summer of 2025, a friend working on AI data collection complained in a group chat that his crawler project had accumulated six dependencies: requests for HTTP, Playwright for browser automation, Scrapy for scheduling, faker-useragent for UA spoofing, and a custom anti-detection script sandwiched in between. Every deployment was a hassle, and version compatibility was a tangled mess.

He said, I just want to grab some data from a webpage, why does it feel like I'm maintaining a distributed system?

I didn't respond at the time, because I didn't have a good answer either. Until last month, I discovered Scrapling, a project with 68,526 Stars, 6,786 Forks, and steadily growing monthly PyPI downloads. What it does is simple: it packages all those tools above into a single framework, from a one-off HTTP request to a full-scale concurrent crawler, all in one library.

Sounds great.

But as someone who has stepped on countless open-source landmines, my first reaction wasn't excitement, but caution. This kind of 'all-in-one' framework is either a truly one-stop solution or a Frankenstein's monster where each layer only performs at 60%. I spent a few days tearing apart its source code, and my conclusion is: it's better than I imagined, but there are also some issues you won't find in the README.

How Fragmented Is the Crawler Toolchain, Really?

Honestly, if you haven't done serious crawler development, you might not understand this pain point. A typical non-trivial crawler project requires at least these components:

Think about it: each of these five components has its own mainstream solution, and there is almost no interoperability between them. Cookies grabbed with requests can't be directly fed to Playwright. Crawler rules written for Scrapy can't be reused for browser automation scenarios. The result is that every new project repeats the work of writing glue code.

Scrapling's solution is to introduce a unified session management layer. In the scrapling/fetchers/ directory, it defines three types of Fetchers: FetcherSession for HTTP requests (using curl_cffi under the hood), DynamicSession for browser automation (using Playwright), and StealthySession for anti-detection browsers (using Patchright, a Playwright fork for anti-detection). Crucially, these three Sessions share the same interface, the same Response object, and the same parser.

This means you can write inside a Spider like this:

class MultiSessionSpider(Spider):
    def configure_sessions(self, manager):
        manager.add("fast", FetcherSession(impersonate="chrome"))
        manager.add("stealth", AsyncStealthySession(headless=True), lazy=True)

    async def parse(self, response: Response):
        for link in response.css('a::attr(href)').getall():
            if "protected" in link:
                yield Request(link, sid="stealth")  # Route to anti-detection browser
            else:
                yield Request(link, sid="fast")      # Route to HTTP request

Within the same Spider, normal pages go through HTTP requests (fast), and pages protected by Cloudflare automatically switch to StealthySession (reliable). The lazy=True parameter in configure_sessions is also clever; it only starts the browser when a request actually needs the stealth channel, consuming no resources otherwise.

Frankly, this design reminds me of database ORMs. The underlying database can be swapped between MySQL, PostgreSQL, and SQLite, but the upper-level code doesn't perceive it. Scrapling does the same thing in the crawling domain, decoupling 'how to get data' from 'how to process data'.

Adaptive Element Tracking: Simpler Than You'd Think

Scrapling's most hyped feature is 'adaptive scraping': when a website redesigns, your selectors can still locate the elements.

I initially thought there was some deep learning model behind this, but digging into the source code revealed that the implementation in scrapling/core/storage.py is surprisingly straightforward. It uses SQLite to store a 'fingerprint' for each element, composed of features like tag name, class list, text content, and attribute sets, and then uses difflib.SequenceMatcher for similarity matching.

The storage layer uses WAL mode (Write-Ahead Logging). Line 76 of scrapling/core/storage.py directly executes PRAGMA journal_mode=WAL, combined with RLock for thread safety. This means even if you read and write element fingerprints simultaneously in a multi-threaded crawler, the table won't lock.

The matching logic is in the find_similar method of scrapling/parser.py. It essentially calculates a similarity score for each candidate element, then sorts and returns them. No machine learning, no vector databases, just simple string similarity with feature weighting.

This actually puts me more at ease. Too many open-source projects use AI as a marketing gimmick; Scrapling doesn't. Its adaptive capability comes from engineering intuition, not chasing trends. The method is simple, but in real-world scenarios, most website redesigns just change class names or adjust DOM hierarchies. Text content and structural relationships remain stable, so SequenceMatcher is perfectly adequate.

But here's a detail: the CSS selector-to-XPath engine in scrapling/core/translator.py is adapted from Parsel (Scrapy's parser), as clearly stated in the file header comments. Scrapling's innovation lies in adaptive storage and element relocation; the parsing engine itself borrows a mature solution.

The Spider Engine: An Underrated Engineering Detail

Actually, I think Scrapling's Spider engine is the part most worth discussing. Not because of flashy features, but because its engineering decisions are very restrained.

The CrawlerEngine class in scrapling/spiders/engine.py uses anyio for async concurrency, rather than asyncio directly. anyio is an abstraction layer compatible with both asyncio and trio, meaning Scrapling's Spider can theoretically run on two async runtimes. Although the documentation currently only shows asyncio usage, this design choice leaves room for future expansion.

The pause/resume mechanism in scrapling/spiders/checkpoint.py uses pickle to serialize the crawler state, including the pending request queue and visited URL set, then atomically writes to a temporary file before replacing the original. The CheckpointManager.save method writes a .tmp file first, then replaces it with the official file, avoiding state corruption from write interruptions. This detail shows the author has genuinely used this in a production environment, not just written demos in an IDE.

The robots.txt parsing in scrapling/spiders/robotstxt.py is also worth mentioning. It supports both Crawl-delay and Request-rate rate-limiting directives, and takes max(user-configured delay, robots.txt required delay) as the final delay. This means if you configure download_delay=0.5, but the target site requires 2 seconds, it will ultimately execute at 2 seconds, rather than overriding with your configuration.

This kind of respect for standards is actually quite rare in crawler frameworks.

Is It Actually Fast? I Laughed After Seeing the Benchmarks

The benchmark table in the README is intimidating: Scrapling's text extraction speed is 2.02ms, crushing BS4's 1584ms.

But look closely, and the truth is that the gap between Scrapling and Parsel/Scrapy is only 2.02ms vs 2.04ms, less than 1%. The crushed opponents (BS4, MechanicalSoup, Selectolax) are known for being slow in the first place. Choosing them for comparison is like racing a bicycle.

Honestly, the biggest problem with this benchmark isn't data falsification, but choosing the wrong reference point. It should be compared against Parsel (since Scrapling's parser is based on Parsel), not BeautifulSoup. Comparing against BS4 only proves Scrapling is faster than BS4, but lxml achieved that a decade ago.

Another benchmark is adaptive element finding: Scrapling 2.39ms vs AutoScraper 12.45ms. This gap is real, because AutoScraper uses rule matching, while Scrapling uses SQLite storage with preprocessing. The data structure advantage is tangible.

I ran through the storage logic in scrapling/core/storage.py. It uses sha256 for element identifier hashing. The _get_hash method in scrapling/core/storage.py also concatenates the identifier length to reduce collision probability. This hashing strategy can still maintain O(1) lookup for millions of stored elements, and is indeed the performance foundation for the adaptive feature.

Some Truths You Need to Know

Scrapling is a solo project.

Contributor data pulled via gh api shows that D4Vinci (Karim Shoair) alone contributed 1,450 commits; the second-place contributor has only 10. Among the 27 contributors, most handle documentation translations and minor bug fixes. This isn't to say solo projects are bad, but a bus factor of 1 means if the author stops maintaining it, the project stops.

The cautionary tale of Camoufox is still there: a browser anti-detection framework that was very popular in 2025 hasn't been updated for over a year by 2026.

Another noteworthy point is the commercialization direction. The README lists 11 Platinum Sponsors, all proxy service providers (Proxidize, ColdProxy, 9Proxy, BirdProxies, etc.). Scrapling's business model is 'free framework + recommended paid proxies,' which differs from the sponsorship model of most open-source projects by embedding itself directly into the crawler industry's supply chain.

This isn't inherently bad, but the implication is that Scrapling's 'anti-detection' capability is highly dependent on proxy services. StealthyFetcher defaults to launching Chromium via Patchright. The browser itself is heavy (starting at several hundred MB of memory), and with the extra wait time required by solve_cloudflare=True, the latency for a single request is much higher than the HTTP Fetcher. The quality of the proxy pool directly determines your crawler's success rate.

The MCP server (scrapling/core/ai.py) is another thing worth watching but still in its early stages. It wraps Scrapling as an MCP tool, allowing AI Agents like Claude and Cursor to directly invoke crawling capabilities. The idea is great, but Issue #366 exposed a fundamental problem: encountering control characters (U+0008) in a page causes the MCP get tool to crash directly. This bug has existed since June and hasn't been fixed in the latest v0.4.10.

Also, the Scrapy integration (scrapling/integrations/scrapy.py) just released in v0.4.10 uses a @scrapling_response decorator to replace Scrapy's Response with Scrapling's Response. This feature has only been live for 4 days, with very little community feedback, so treating it as a stable feature is premature.

What I Learned from Scrapling: An Engineering Pattern Called 'Unified Session Layer'

After tearing down Scrapling, I kept pondering one question: what is its most valuable part?

It's not the adaptive parsing, not the anti-detection browser, and not the Spider engine. Taken individually, each of these has more specialized alternatives.

What Scrapling truly excels at is designing a 'Unified Session Layer.' The core idea of this layer is: different data acquisition channels (HTTP, browser, anti-detection browser) should share the same interface, and the choice of channel should be a runtime decision, not an architectural one.

I call this pattern the Session Dispatcher Pattern. Its core elements are:

  1. All Fetchers implement the same interface (fetch, get, post), returning the same Response type.
  2. Channel selection is dynamically routed at runtime via sid (Session ID), rather than being hardcoded.
  3. Channel creation can be lazy-loaded (lazy=True), initializing only when truly needed.

This pattern wasn't invented by Scrapling; similar abstractions have existed in HTTP clients for a while (like httpx's Client vs AsyncClient). But applying it to the entire crawler toolchain, unifying everything from HTTP to browsers to MCP, is something Scrapling is the first to seriously do.

If you're working on a crawler project, even if you don't use Scrapling, this pattern is worth borrowing. Write your own SessionManager class, encapsulating requests, playwright, and anti-detection logic, uniformly returning your defined Response protocol. It might seem superfluous at first, but when you need to switch a request from HTTP to browser mode, changing one sid parameter versus rewriting the entire crawler logic will make you grateful for the foresight.

For Scrapling itself, my selection advice is: suitable for small to medium-sized crawler projects, especially scenarios requiring frequent switching between HTTP and browser modes. If your project is purely API scraping (no JS rendering needed), httpx is enough. If your project is an industrial-scale crawler with millions of URLs, Scrapy's scheduling and middleware ecosystem is more mature. Scrapling's sweet spot is that middle layer, where you need flexibility but haven't scaled to the point of needing a self-built scheduling cluster.

68K Stars didn't come from nowhere; this project solves a real problem. But remember, it's currently maintained by one person and is highly dependent on the proxy service ecosystem. If you decide to use it, be prepared for vendor lock-in to a specific proxy provider, or at least abstract your proxy layer so it's not too tightly coupled with Scrapling's Session.