Why Clicking 'Offline' in Nacos Still Routes Traffic to a Dead Instance
Every rolling deployment that skips these steps produces user-facing `Connection Refused` or `Read Timeout` errors during the release window. The three-part pattern — PreStop deregistration, client retry on connect failure, and graceful shutdown — eliminates those errors without slowing down deployments, and the retry safety argument (no business logic executed) makes it auditable for compliance-sensitive systems.
The gap between a Nacos instance going offline and every gateway learning about it is a cache-propagation delay, not a registry bug. Gateways and microservices using Ribbon or Spring Cloud LoadBalancer poll for updates on a timer; UDP push notifications can be lost in production networks. During that window, requests still hit the old IP, and if the process is already killed, the OS returns a RST packet and the caller gets a `Connection Refused`.
Small teams often insert a `sleep 40` into their CI/CD pipeline after calling the Nacos deregistration API, letting the machine keep serving while caches converge. That approach works but makes rolling a 500-node service take hours and offers zero protection against sudden machine failures.
A production-grade lossless shutdown ties three mechanisms together: a K8s PreStop hook that deregisters the Pod from Nacos before termination, client-side retry in the load balancer that catches `ConnectException` at the TCP handshake stage and silently retries another IP, and Spring Boot's graceful shutdown that drains in-flight requests before the process exits. The retry is safe because a connection refused at the TCP layer means no application logic ever ran, making the operation effectively idempotent.
The `sleep 40` pattern is a surprisingly common interview answer, which suggests many engineers treat the registry as real-time and never trace the cache-update path end to end.
The safety of the client retry hinges on a low-level TCP detail: a `ConnectException` means the three-way handshake never completed, so the HTTP request body was never transmitted and no business side effect occurred. That turns a network error into a free retry.
Binding the deregistration call to the K8s PreStop hook rather than an external CI script makes the shutdown self-contained; the Pod itself is responsible for leaving the pool cleanly, which also handles unplanned scale-down events that a Jenkins pipeline would never see.