跪拜 Guibai
← Back to the summary

Why Clicking 'Offline' in Nacos Still Routes Traffic to a Dead Instance

A couple of days ago, I asked a very routine interview question:

**❝**When you release a new version online, how do you ensure that user requests don't fail when the old service goes offline?

He answered confidently: That's simple. Go to the Nacos console, find that instance, and click the offline button. Or just run kill -15 in the release script to kill the process. Nacos won't receive a heartbeat and will naturally remove the node, so the gateway won't send traffic to that machine anymore.

Well...

In a real production environment, if the release pipeline really only does this, then during the one or two minutes of each release, your gateway logs will definitely be flooded with a bunch of Connection Refused or Read Timeout errors.

If this happens to coincide with the evening peak hours, the user's direct experience is: clicking a button, and the interface pops up a glaring network error.

1. Why Can't the Traffic Be Stopped?

Many students have a misunderstanding about the registry center, thinking it is an absolutely real-time system: as soon as Nacos knows a node is offline, all dependent microservices instantly know it too.

But in a distributed system, information transmission takes time.

A service caller, such as a Gateway, needs to know the target service's IP before initiating an HTTP request. But for performance reasons, the gateway cannot query Nacos for every single request.

It uses client-side load balancing components like Ribbon or SpringCloud LoadBalancer, which maintain a ServerList service IP list cache in their own memory.

The pitfall lies precisely in the update mechanism of this cache:

  1. The caller starts a background scheduled task by default, pulling the latest instance list from Nacos at regular intervals.
  2. Although the Nacos server also has a UDP mechanism for pushing changes in real time, UDP pushes can be lost in complex production networks.
  3. This results in a time gap of up to tens of seconds.

During these tens of seconds, although the Nacos server has already marked that node as offline, the gateway's local cache hasn't been refreshed yet, and the gateway is still holding the old list.

Now recall what you did after clicking offline in the console:

You triggered the offline action, and immediately after that, you triggered a restart (the process was killed). But at this moment, the gateway hasn't updated its cache yet and is still sending user requests to this already dead machine.

The operating system sees that no one is listening on the port, directly returns an RST packet, and the gateway immediately reports Connection Refused.

2. The Solution for Small and Medium Teams

Knowing the cause, the solution becomes clear. Many small and medium teams do this: add a line of sleep 40 in the CI/CD release pipeline.

The process becomes:

  1. Actively drain traffic: the script first calls Nacos's OpenAPI to mark the machine as offline.
  2. The script forces a sleep 40 seconds. During these 40 seconds, the machine is still running normally and processing requests; we are just waiting for all gateways across the network to update their caches.
  3. Graceful shutdown: After the 40 seconds pass, no one is sending requests here anymore, and then we send kill -15 to kill the process.

This solution does work. But if you take this solution to an interview at a big tech company, it's still a failing grade. The reason is simple: it's too slow and not universally applicable.

If a core service has 500 nodes, a rolling release where each node has to wait 40 seconds would take several hours for a single release.

Moreover, simply relying on waiting cannot handle sudden situations like a physical machine crashing or network jitter, because when a machine crashes unexpectedly, you have no chance to sleep.

3. The Lossless Offline Solution of Big Tech Companies

Step 1: Orchestration Layer Traffic Draining

In a more formal company, the offline action should not be controlled by a Jenkins script, but should be deeply bound to the K8s Pod lifecycle.

When K8s decides to scale down or update a Pod, it does not immediately send kill -15. Instead, it first triggers a lifecycle pre-hook, the PreStop Hook. In the K8s yaml, you write it like this:

lifecycle:
  preStop:
    exec:
      command:
        - /bin/sh
        - -c
        - |
          # 1. Immediately send an offline request to Nacos to actively deregister itself
          curl -X PUT "http://nacos-server:8848/nacos/v1/ns/instance?serviceName=my-service&ip=${POD_IP}&port=8080&enabled=false"
          # 2. A symbolic short pause of 3~5 seconds to give Nacos's UDP push a little time
          sleep 5

This step encapsulates the active offline logic inside the container. Whenever a Pod is about to die, the first thing it does before dying is to go to the registry center to "cancel its account."

Step 2: Client-Side Retry

Even with PreStop, it's still impossible to 100% avoid residual requests hitting the gateway's old cache during those few seconds.

When this machine stops accepting new requests, requests sent by the gateway will immediately receive an operating system Connection Refused. Note this extremely crucial detail: ConnectException occurs during the TCP three-way handshake phase, at which point the HTTP business data has not been sent out at all!

Since the business logic hasn't been executed, this request is absolutely safe and absolutely idempotent.

Therefore, we must enable the underlying retry mechanism in the gateway or upstream microservice. Taking SpringCloud as an example, just add:

spring:
  cloud:
    loadbalancer:
      retry:
        enabled: true # Enable client-side retry

The miracle that happens at this moment is:

The gateway sends a request using the old IP -> hits a wall with Connection Refused -> the gateway's underlying LoadBalancer catches the connection exception, silently suppresses it, and then automatically picks another healthy IP from the cache to re-initiate the request.

The entire process completes within a few milliseconds. What the end user sees on their phone is a normal 200 OK; the user is unaware.

Step 3: Application Graceful Shutdown

With the previous two steps, new requests entering this machine have been perfectly blocked and retried. The last step is to handle those old requests that have already been accepted and are currently being processed.

In the Spring Boot project's application.yml, add these two lines of configuration:

server:
  shutdown: graceful # Enable graceful shutdown
spring:
  lifecycle:
    timeout-per-shutdown-phase: 30s # Wait up to 30 seconds for old requests

Cooperating with K8s's shutdown mechanism, the Java process waits for a period of time, then completely releases resources.

4. The Complete Lossless Offline Process

To make it more intuitive for you, I drew this linkage sequence diagram covering K8s + Nacos + Client Retry.