跪拜 Guibai
← All articles
Frontend · Android · Flutter

Deslop Moves Flutter Duplicate Detection into the AI Agent’s Write Loop

By 恋猫de小郭 ·
Read original on juejin.cn ↗ Google Translate ↗ Alt translation

AI coding agents produce structurally identical code under different names at scale, and Flutter’s declarative widget trees amplify the problem. Deslop shifts duplicate detection from post-commit linting to a pre-generation retrieval step, cutting the review burden before near-clones diverge and accumulate separate bug-fix histories.

Summary

AI-generated Flutter code accumulates near-identical loading branches, formatters, and repository logic because LLMs lack a global view of the codebase. Deslop attacks this by parsing Dart syntax trees with Tree-sitter, normalizing identifiers and literals, then computing Merkle hashes for exact clones and MinHash/LSH signatures for approximate clones. An optional embedding layer can catch semantically similar but syntactically different implementations.

The tool ships as a CLI, a VS Code extension with an LSP, and an MCP server that lets coding agents query the repository’s structure index before generating a new function. A fused similarity score — the max of structural match, token Jaccard, and embedding cosine — drives clustering and ranking, with a CI gate that fails the pipeline when duplication exceeds a configured threshold.

Deslop does not demand that all duplicates be eliminated. Identical AST shapes can serve different domain semantics, and Flutter projects carry legitimate repetition across platform isolates, diverging page skeletons, and independent packages. The tool supplies structural evidence; the developer still decides what to merge.

Takeaways
Deslop normalizes Dart ASTs by replacing all identifiers with __ident__ and all literals with __literal__, so renamed variables and reformatted code produce the same structural fingerprint.
Type-1 and Type-2 clones are caught via BLAKE3 Merkle hashes on subtrees of at least 30 AST nodes; Type-3 clones use MinHash signatures split into 32 bands with a token_jaccard threshold of 0.90 and a minimum of 40 AST nodes per side.
An optional embedding layer (Ollama, nomic-embed-text, off by default) detects Type-4 clones — behaviorally similar code with different syntax, such as imperative loops vs. functional collection operations.
Candidate pairs are fused by taking the maximum of structural (0/1), token_jaccard, and embedding_cos scores; pairs reaching 0.85 enter transitive clusters where A–B and B–C similarity groups A and C together.
Cluster ranking multiplies clone_node_count × (cluster_size − 1) × log2(1 + spanned_bytes), prioritizing high-impact duplicates while dampening the weight of single large generated files.
The MCP server queries the running LSP over Unix sockets or TCP loopback, so an agent calling find-similar gets deterministic results without re-scanning the whole repo.
A CI GitHub Action fails the pipeline (exit code 3) when duplication exceeds a threshold in .deslop.toml, while still producing JSON/TXT/HTML reports for inspection.
Official AGENTS.md rules instruct the agent to call find-similar before creating any function, class, fixture, parser, route, or ViewModel longer than a few lines, with fused ≥ 0.85 meaning reuse-or-extract and fused < 0.6 meaning proceed.
Legitimate duplication in Flutter includes platform isolates, diverging page skeletons, domain-specific repositories with coincidental structural matches, deliberately expanded test cases, performance-sensitive dedicated loops, and independently published packages.
Conclusions

Deslop’s architecture inverts the usual static-analysis workflow: instead of scanning after code lands, it injects a retrieval step into the agent’s generation loop, making the LSP the live index and the MCP server a thin query layer over IPC.

The transitive-closure clustering is a deliberate trade-off — it increases recall at the cost of precision, meaning a cluster can contain members that are not directly similar to each other. This forces a human to inspect before extracting a shared abstraction.

The fused score uses max() rather than a weighted average, which means a pair with perfect structural match (1.0) passes the 0.85 gate even if token and embedding signals are weak — a design choice that prioritizes not missing exact clones over suppressing false positives.

Deslop explicitly down-weights structural_only matches and large data blocks to 0.15×, acknowledging that identical AST shapes with thin content (e.g., empty widget shells) are noise, not actionable duplication.

The tool’s own guidance draws a sharp line: structural similarity is evidence, not a mandate. Two functions with identical AST shapes but different domain semantics (price formatting vs. weight formatting) should stay separate, and merging them would create a fragile abstraction that breaks under future requirements.

Concepts & terms
Tree-sitter AST normalization
Deslop parses Dart into a concrete syntax tree, then replaces all identifiers with __ident__ and all literals with __literal__, stripping comments and whitespace. This makes the tree represent syntactic shape alone, so renamed variables and reformatted code produce identical normalized structures.
Merkle Hash (BLAKE3) for clone detection
A bottom-up hash of AST subtrees (minimum 30 nodes) where each node’s hash depends on its type and the ordered hashes of its children. Identical subtrees produce identical fingerprints, catching Type-1 and Type-2 clones regardless of identifier names.
MinHash and LSH for approximate clones
Normalized AST node types are grouped into k-grams of width 5, then hashed into a 128-value MinHash signature split into 32 bands of 4 rows each. A band collision between two code segments triggers a full Jaccard similarity estimate, catching Type-3 clones where a few statements differ.
Transitive closure clustering
If code pair A–B and pair B–C both exceed the fused similarity threshold, A, B, and C are placed in the same cluster even if A–C similarity is below threshold. This increases recall but means not all members of a cluster are equally similar.
Fused similarity score
The maximum of three independent signals: structural (0 or 1 for exact AST match), token_jaccard (approximate syntactic sequence similarity), and embedding_cos (semantic vector cosine similarity). A pair is retained when max ≥ 0.85.
MCP server over LSP IPC
Deslop’s MCP server does not re-analyze the repository. It queries the running VS Code LSP over Unix sockets (macOS/Linux) or TCP loopback with a token (Windows), returning cached structural-index results to coding agents with low latency.
Source: juejin.cn ↗ Google Translate ↗ Backup ↗