---

theme: smartblue

Nowadays, almost everyone uses AI to write code, but in practice AI often lacks global awareness. Even if you write rules, it may not follow them. For example, I saw a post from a netizen yesterday that felt quite typical:

In this situation, if you don't review carefully, the project may unknowingly accumulate a large batch of similar code — identical Loading, Error, Snackbar, date formatting, and Repository conversion logic.

Then Deslop is used in this area. It parses the syntactic structure of Dart code, then finds identical code, code that is identical after variable renaming, and code that is similar after partial modifications through rules. At the same time, it provides a query of "whether there is a similar implementation in the repository" through MCP before the Agent writes code.

Yes, the most important part is providing this query.

It moves duplicate detection from post-commit quality checks into the Agent's generation loop. For example, similar code:

In fact, if you directly ask AI to judge these two implementations, AI may not even be able to determine whether they are duplicates, because the function names, parameter types, and variable names are all different. Eventually, the two implementations may start evolving independently, and when fixing empty strings, internationalization, or name ordering, it's easy to only change one of them.

Flutter projects are especially prone to this situation, because the declarative Widget tree contains a large number of stable structures, and AI can easily generate the following content repeatedly:

• AsyncValue, FutureBuilder, or loading/error/data branches in custom states
• Scaffold, SafeArea, empty states, and error cards
• Field mapping from DTO to Entity
• Similar retry, exception conversion, and pagination logic in Repositories
• Formatting functions for dates, amounts, file sizes, etc.
• Device configurations and Provider overrides for Golden Tests
• Wrapper functions for Snackbar, Dialog, BottomSheet

Sometimes these pieces of code are only a dozen lines, but after months of continuous generation, each basic capability may end up with three or four versions.

Deslop's approach is a bit interesting. It uses Tree-sitter to parse Dart's syntax tree, then goes through normalization, fingerprint calculation, approximate matching, clustering, and ranking. The official complete process is:

discover → parse → normalize → fingerprint → cluster
         → LSH → embed → fuse → rank → render

Here, the first layer first restores the code to a syntactic structure. Tree-sitter parses functions, calls, conditions, property accesses, loops, and expressions into an AST, then Deslop normalizes the syntax tree:

• Identifiers are uniformly replaced with __ident__
• String, number, and character constants are replaced with __literal__
• Comments, whitespace, and other syntactic trivia are removed

After processing, the previous formatUserLabel and buildAuthorName both become the same structure of "function declaration, one parameter, string interpolation, two property accesses". Simply changing function names and variable names can also be detected in this situation.

So Deslop's core focus is the syntactic shape of the code. Variable renaming and formatting will not break the results.

Then the second layer uses Merkle Hash to find completely identical subtrees. By default, Deslop processes subtrees with no fewer than 30 AST nodes, calculating BLAKE3 Merkle Hash from leaves upward. If node types and child order are the same, they will ultimately get the same fingerprint. This part mainly captures Type-1 and Type-2 Clones:

Type	Typical Situation	Deslop's Identification Method
Type-1	Code, variable names, constants all identical	AST Merkle Hash
Type-2	Variable names, function names, or constants different	Normalize AST then calculate Hash
Type-3	A few statements added or deleted, main body still similar	AST k-gram, MinHash, and LSH
Type-4	Writing style obviously different, behavior close	Optional code Embedding

Deslop also creates window fingerprints for 2 to 8 consecutive sibling statements, so even if the outer structures of two function segments are different, as long as there is a set of consecutive repeated statements in the middle, there is a chance to be discovered.

Next, the third layer uses MinHash and LSH to find "approximate copies". Completely identical Hash cannot handle situations like locally adding a judgment or adjusting some statements, so Deslop composes the normalized AST node types into k-grams of width 5, then calculates a MinHash signature containing 128 values, and splits it into 32 bands, each band containing 4 rows.

The purpose of this design is to quickly recall possibly similar code pairs. After a band collision occurs, Deslop estimates Jaccard similarity based on the complete signature. When relying solely on this signal path, the current rules also require:

• token_jaccard ≥ 0.90
• Both ends have at least 40 AST nodes

The purpose of this is to reduce noise brought by ordinary Flutter Widget structures. Many Widgets have build → Column → children. Judging duplication based on just a few identical nodes would quickly make the report lose value.

Finally, there is optional semantic Embedding. Deslop also supports finding implementations with "close behavior, large syntactic differences" through code vectors, such as imperative loops and functional collection operations.

However, this layer is turned off by default. Currently, only the Ollama Provider is implemented, with the default model being nomic-embed-text. It needs to be explicitly enabled with --embeddings auto or --embeddings required.

Finally, the three scores are merged. Each candidate code pair has three independent signals:

• structural: Complete structural match, value is 0 or 1
• token_jaccard: Approximate syntactic sequence similarity
• embedding_cos: Code vector cosine similarity

The fusion value in the candidate stage adopts the maximum of the three max(structural, token_jaccard, embedding_cos). It is retained only when reaching FUSED_THRESHOLD = 0.85. Then Clusters are built through transitive closure:

If A is similar to B, and B is similar to C, the three enter the same group, even if A and C do not directly reach the threshold.

This clustering method expands the recall range, but of course it also brings a problem that requires manual attention:

Inside a Cluster, it is not guaranteed that any two members are equally similar. Larger Clusters may be linked together by intermediate code. You cannot directly extract a common base class just by seeing a set of results.

Deslop also applies a content gate to results that are structurally identical but have insufficient content evidence, and by default reduces the weight of structural_only and large-block data-type duplicates to 0.15 times.

Then there is ranking. Deslop uses the following formula to rank Clusters:

weight =
  clone_node_count
  × (cluster_size − 1)
  × log2(1 + spanned_bytes)

It mainly considers three dimensions:

• How many AST nodes the duplicate segment contains
• How many copies of the same segment appear
• How many bytes these codes occupy in the source code

For example:

• cluster_size − 1 can be roughly understood as "how many copies can be deleted"
• log2 suppresses the volume advantage of very large files, preventing a single huge generated file from occupying the entire leaderboard
• The final report prioritizes displaying the duplicate items with the highest benefit

Then Deslop currently also provides several forms. The forms share the same deslop-core analysis engine, but the usage scenarios differ greatly:

• CLI is suitable for initial audits, CI, and cold scans. After executing deslop ., it will generate JSON, TXT, and HTML reports in the project's .deslop/ directory by default. JSON is for Agent and automation processing, TXT is suitable for the terminal, and HTML is for manual viewing.

• The VS Code extension starts an LSP, listens for file changes and incrementally updates results. Each file uses content Hash as the cache key, and files without changes can skip analysis.

• MCP provides a query entry for Coding Agents. The architectural details here are very interesting:

  • deslop-mcp itself does not re-analyze the entire repository. It queries the latest results from the running LSP through local IPC.
  • macOS and Linux use Unix Sockets, Windows uses local TCP Loopback with a Token.
  • When the Agent calls find-similar once, it usually does not need to re-traverse all Dart files.

• CI is responsible for long-term trends. When the duplication rate exceeds the threshold in .deslop.toml, the CLI returns exit code 3. The report is still generated, making it convenient to view specific issues from the failed Pipeline.

name: deslop
on: [push, pull_request]

jobs:
  duplication-gate:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: Nimblesite/[email protected]
        with:
          fail-over: "5.0"   # or omit to use [threshold] in .deslop.toml

If you only run deslop ., its differences from other static analyzers are mainly reflected in Dart support, AST structure detection, and ranking. But supporting MCP is different. It can be brought into the Agent's write loop, for example:

Agent prepares to write a function
        ↓
Call find-similar
        ↓
Query the current repository's structure index
        ↓
Find approximate implementation
        ↓
Reuse, extend, or abandon new creation

This is equivalent to adding a set of local code retrieval for the Agent. The LLM is responsible for understanding requirements, and Deslop is responsible for answering whether there is already a similar structure in the repository. The retrieval results are deterministic AST, Token, and optional vector analysis. There is no need to let the model blindly grep around first, then guess which function might be relevant. This is theoretically less prone to hallucination.

However, installing MCP does not guarantee that the Agent will actively call it, so the official team specifically provides a section of AGENTS.md/CLAUDE.md rules, requiring the Agent to call find-similar before creating new functions, classes, Fixtures, Parsers, Routes, or ViewModels that exceed a few lines.

The official currently recommended judgment intervals are:

• fused ≥ 0.85: Prioritize reuse or extraction
• 0.6 ≤ fused < 0.85: Judge after reading candidate code
• fused < 0.6: Structural distance is far, can continue creating

The author also packaged Deslop into a more cautious Dart/Flutter Agent Skill:

First read-only scan, then user confirmation, then judge whether each duplicate item is worth merging, and finally verify through tests before and after refactoring: https://github.com/kevmoo/kevmoo_skills/blob/main/skills/deslop-duplication-audit/SKILL.md

But it's not that all duplicate code should disappear. For example, two functions can have the same structure while bearing completely different domain semantics. For instance, price formatting and weight formatting may currently both be:

value.toStringAsFixed(2)

If they are merged into formatNumber() here, it will actually erase domain meaning. Later, when price needs currency precision and weight needs unit conversion, this common function will be split apart again.

In fact, there are several common types of reasonable duplication in Flutter:

• iOS and Android platform implementations need to remain isolated
• Multiple pages follow the same Widget skeleton, but lifecycles and interactions are diverging
• Repositories in different Domains happen to adopt the same process
• Test cases deliberately expand Arrange/Act/Assert for independent reading
• Performance-sensitive loops maintain dedicated implementations to avoid closures and dynamic dispatch
• Different packages need independent publishing, and forced sharing would create reverse dependencies

So structural duplicate checking only provides primary evidence. Architectural boundaries still need to be judged by the developer themselves.

Links

https://apparencekit.dev/blog/deslop-duplicate-code-flutter/