Microservices Done Right: Splitting, Communicating, and Governing Without Creating a Distributed Monolith

1. Introduction: Why Microservices

Monolithic applications are the most efficient choice in the early stages of a business: one repository, one build, one deployment, low communication overhead for the team, and a clear debugging path. But when business scale, team size, and iteration frequency all grow at once, the costs of the monolith start to show:

• Build and deployment coupling: any change to a line of code requires a full rebuild and full release, and the scope of regression testing grows linearly with the code size.
• Technology stack lock-in: the entire system is forced to use the same language, framework, and middleware versions.
• Failure propagation: a memory leak in a non-core module can bring down the entire process.
• Team collaboration bottleneck: multiple teams working in parallel on the same codebase makes merge conflicts and release queueing the norm.

Microservices alleviate these problems by splitting along business boundaries, deploying independently, and evolving independently. But one thing needs to be clear: microservices are not a free lunch. They trade "code complexity" for "distributed systems complexity." This article follows the order of "decomposition → communication → governance → data consistency" to lay out a workable design methodology.

The diagram below summarizes the overall thread of this article; the subsequent chapters follow this order:

flowchart LR
  A["Service decomposition"] --> B["Service communication"]
  B --> C["Service governance"]
  C --> D["Data consistency"]
  A -.-> A1["Single responsibility / independent deployment / decentralization"]
  B -.-> B1["Sync vs async / gateway and BFF"]
  C -.-> C1["Registration discovery / config / circuit breaking / observability"]
  D -.-> D1["Eventual consistency / idempotency / compensation"]

2. Core Principles of Service Decomposition

2.1 Single Responsibility: Split by Business Capability, Not by Technical Layer

The most common mistake is splitting by technical layer (Controller service, Service service, DAO service). This causes every business change to require releases across multiple services, amplifying coupling instead of reducing it.

The right approach is to split by business capability, referring to the Bounded Context in Domain-Driven Design (DDD):

Splitting approach	Example	Evaluation
By technical layer	User Controller service / User DAO service	❌ One requirement touches multiple services
By business capability	User service / Order service / Inventory service	✅ Change cohesion
By resource type	Image service / Video service	⚠️ Needs case-by-case business judgment

A simple standard to judge whether a service has a "single responsibility": do all the reasons for changing this service belong to the same business domain? If two features are always modified for different reasons, they should be split; if they are always modified together, they should be merged.

The evolution path from monolith to microservices can be understood like this:

flowchart TD
  M["Monolithic application"] --> S1["Identify bounded contexts by business capability"]
  S1 --> S2["Split out 3~8 core domain services"]
  S2 --> S3["Independent deployment and independent data"]
  S3 --> C{"Still need to release together?"}
  C -->|"Yes"| W["Distributed monolith: boundary was wrong, must re-divide"]
  C -->|"No"| OK["Boundaries are clear, refine as needed"]

2.2 Independent Deployment: Splitting Only Counts When You Can Deliver Independently

A service is truly "independent" only if it has the following capabilities:

• Independent data storage: does not share database tables with other services, avoiding implicit coupling.
• Independent build artifact and release pipeline: can be released alone without affecting other services.
• Independent runtime resources: CPU, memory, connection pools do not compete with each other.
• Backward-compatible interface contracts: consumers do not need to be released in lockstep when upgrading.

If two services still must "ship together" after the split, the decomposition only added operational cost without bringing any benefit.

2.3 Decentralization: Devolving Data, Governance, and Decision-Making

Decentralization covers three levels:

1. Data decentralization: each service owns its data and exposes it via APIs or events, rather than letting other services connect directly to its database.
2. Governance decentralization: teams choose their own technology stacks and implementation details; the platform team only provides standard infrastructure (registry center, config center, observability).
3. Decision decentralization: the team that owns a service decides its internal implementation details; only contracts are agreed across services.

Note: decentralization does not mean "no standards." Cross-service contracts like interface protocols, error code conventions, log formats, and trace instrumentation must be unified, or governance costs will rise sharply.

2.4 Splitting Granularity: From Coarse to Fine, Avoid Over-Splitting

There is no standard answer for splitting granularity, but there are some pieces of experience:

• Coarse first, fine later: start with 3–8 core domains, then refine as the team and business grow.
• Conway's law: system architecture tends to mirror organizational communication structures; splits should align with team boundaries.
• One service, one team: if a single service needs three teams to maintain it, the boundary is wrong.
• Beware of the distributed monolith: services are split, but every requirement still means changing five services and releasing them together. That is a "distributed monolith," harder to maintain than a monolith.

3. Service Communication Patterns

3.1 Synchronous vs Asynchronous

Dimension	Synchronous (HTTP/gRPC)	Asynchronous (MQ/events)
Coupling	Time-coupled; caller waits	Time-decoupled; producer and consumer do not block each other
Real-time	Strong; immediate result	Weak; eventual consistency
Failure impact	Downstream failures propagate upstream	Downstream failures can be buffered and retried
Complexity	Clear chain, easy to debug	Need to handle idempotency, ordering, duplicate consumption
Use cases	Queries, strong-consistency validation	Notifications, state transitions, cross-domain events

Practical recommendations:

• Queries use sync; commands can be async: reads usually need immediate results; writes that allow eventual consistency should prefer async.
• Core paths use sync; side paths use async: placing an order and deducting inventory is the core path; sending SMS, adding points, and writing audit logs should be async.
• Synchronous calls must set timeouts: a synchronous call without a timeout is a failure amplifier.

The choice between sync and async can be decided by the following path:

flowchart TD
  Q["One inter-service call"] --> R{"Does the caller need an immediate result?"}
  R -->|"Yes"| S["Synchronous HTTP/gRPC"]
  R -->|"No"| T{"Is it part of the core transaction path?"}
  T -->|"Yes"| U["Sync + timeout + circuit breaker"]
  T -->|"No"| V["Async MQ/event"]
  S --> W["Must set timeout and retry backoff"]
  V --> X["Consumer guarantees idempotency and ordering"]

3.2 API Gateway and BFF

An API gateway sits at the system's entry point and handles cross-cutting concerns:

• Routing and protocol conversion
• Unified authentication, rate limiting, circuit breaking
• Canary release and traffic tagging
• Access logging and metrics collection

BFF (Backend for Frontend) is an aggregation layer after the gateway that tailors APIs for specific frontends (Web, App, mini-program):

Web frontend ──┐
App frontend ──┼──> API gateway ──> BFF(Web) ──┐
Mini-program ──┘                BFF(App) ──┼──> User service / Order service / Product service
                                        └──> ...

The value of BFF:

• Aggregation and trimming: one request aggregates multiple downstream services, reducing the number of frontend requests.
• Endpoint adaptation: different endpoints need different fields, avoiding backend interfaces bloating to support multiple clients.
• Change isolation: frontend requirement changes affect only the BFF, not core services.

The cost is an extra layer of calls and operational overhead. Small teams or few client types do not need to force a BFF in.

Let's replace the above textual diagram with a flowchart to make the request path clearer:

flowchart LR
  W["Web frontend"] --> G["API gateway"]
  A["App frontend"] --> G
  M["Mini-program"] --> G
  G --> BW["BFF(Web)"]
  G --> BA["BFF(App)"]
  BW --> U["User service"]
  BW --> O["Order service"]
  BA --> U
  BA --> P["Product service"]

3.3 Protocol Selection

• HTTP/REST: good universality, easy to debug; suitable for external APIs and cross-team interfaces.
• gRPC: based on HTTP/2 and Protobuf, high performance, strong typed contracts; suitable for high-frequency internal calls.
• Message queues: Kafka, RocketMQ, RabbitMQ; suitable for event-driven patterns and peak shaving.

4. Service Governance

4.1 Service Registration and Discovery

After a service instance starts, it registers its address with a registry center. Callers address the service by name rather than by IP:

Service provider ──register──> Registry center (Nacos / Consul / Eureka)
                          │
Caller ──subscribe/pull────┘
   │
   └──> Load balancing selects an instance ──> make the call

Key design points:

• Health check: distinguish "registered and alive" from "business ready" to avoid routing traffic to instances that have not finished initialization.
• Graceful online/offline: remove the instance from the registry before shutdown, wait for in-flight requests to finish, then close the process.
• Local cache: when the registry is unavailable, callers should be able to keep working with a locally cached service list.

The complete flow of a service call in the registration/discovery system:

sequenceDiagram
  participant P as "Service provider"
  participant R as "Registry center"
  participant C as "Caller"
  participant L as "Load balancer"
  P->>R: "Register instance address at startup"
  R-->>P: "Heartbeat and health check"
  C->>R: "Subscribe to service list"
  R-->>C: "Push instance list (local cache)"
  C->>L: "Request by service name"
  L->>P: "Select healthy instance and call"
  P-->>C: "Return result"
  P->>R: "Remove before shutdown"

4.2 Configuration Management

Config centers (Nacos Config, Apollo, Spring Cloud Config) solve three types of problems:

1. Environment isolation: dev/test/prod configs are separated to avoid accidental misuse.
2. Dynamic updates: rate-limit thresholds, feature switches, timeouts can be adjusted without restarting the service.
3. Change audit: who changed what config and when can be traced and rolled back.

Practical points:

• Config changes must support gray push: push to one instance first, observe, then roll out to all.
• Sensitive config (database passwords, secrets) should be encrypted, not left in plaintext in a config center.
• Config items need default fallback values so services can still start if the config center goes down.

4.3 Circuit Breaking and Degradation

A circuit breaker is a state machine:

stateDiagram-v2
    [*] --> Closed
    Closed --> Open: "Error rate exceeds threshold"
    Open --> HalfOpen: "Wait window ends"
    HalfOpen --> Closed: "Probe request succeeds"
    HalfOpen --> Open: "Probe request fails"

• Closed: requests flow normally; error rate is tracked.
• Open: requests are rejected immediately, fail fast, and avoid an avalanche.
• Half-Open: a small number of probe requests are let through; if they succeed, the circuit closes; if they fail, it stays open.

Degradation strategies usually include:

• Return fallback data: if a recommendation service is unavailable, return a popular list.
• Feature degradation: disable non-core features (e.g., product reviews) to keep the main flow working.
• Rate limiting: reject requests above a threshold to protect downstream systems.

Alongside this are timeout control and retry policy: retries must use backoff and a max count, and should only be applied to idempotent operations; otherwise they amplify failures.

4.4 Observability

The premise of governance is "being able to see." Three pillars are all required:

• Logging: structured logs with a unified traceId across calls.
• Metrics: QPS, latency percentiles (P95/P99), error rates, thread pool and connection pool water levels.
• Tracing: cross-service call chains to locate slow calls and abnormal nodes.

5. Distributed Transactions and Eventual Consistency

5.1 Why Distributed Transactions Are Needed

After splitting services, a single business operation can span multiple service databases. For example, "place an order" needs to write to the order database, deduct inventory, and deduct balance. Local transactions cannot cover cross-database operations, so consistency problems arise.

5.2 Comparison of Common Approaches

Approach	Consistency	Performance	Invasiveness	Use cases
2PC/XA	Strong	Low	High	Few internal strong-consistency scenarios
TCC	Eventual	Medium	High	Core transactions like money and inventory
Saga	Eventual	High	Medium	Long-running, multi-step business
Local message table	Eventual	High	Low	Async notification scenarios
Transactional message (RocketMQ)	Eventual	High	Low	Scenarios deeply integrated with MQ

5.3 Practice: Local Message Table + Scheduled Compensation

This is the lowest-cost and most commonly used eventual-consistency approach:

1. Local transaction: business data + message record are written in the same local transaction
2. Async delivery: a scheduled task scans unsent messages and delivers them to MQ
3. Consumption: downstream consumes and guarantees idempotency
4. Status write-back: after successful delivery, update the message status to sent
5. Failure compensation: messages exceeding the max retry count go to a dead-letter queue for manual intervention

Key points:

• The message table and business table are in the same transaction, guaranteeing that "if business succeeds, the message is persisted."
• The consumer must be idempotent: use a business unique key or a deduplication table to avoid duplicate consumption causing duplicate deductions.
• Compensation and alerting are mandatory: eventual consistency relies on "eventually the retry will succeed," so a dead-letter queue and monitoring alerts cannot be omitted.

5.4 Three Baseline Rules for Consistency Design

1. Do not sacrifice availability for strong consistency: most business scenarios can tolerate second-level eventual consistency.
2. Idempotency is the premise of consistency: any write operation that may be retried must be idempotent.
3. Reconciliation as a backstop: no matter how good compensation is, something can still be missed; core money-related business must have a scheduled reconciliation mechanism.

6. Summary

The core of microservices architecture design is not "how finely you split," but clear boundaries, stable contracts, and solid governance:

• Decomposition: split by business capability, guarantee independent deployment, avoid the distributed monolith.
• Communication: queries sync, commands async, gateway as the unified entry, BFF adapting per endpoint.
• Governance: registration and discovery ensure reachability, config management ensures adjustability, circuit breaking and degradation ensure availability, observability ensures inspectability.
• Consistency: prefer eventual consistency, idempotency and compensation are required capabilities, reconciliation backstops core business.

Architecture evolution has no end. Start with the smallest feasible split, and let monitoring data drive the next optimization, rather than designing a "perfect architecture" all at once.