Redis Beyond the Cache: Data Structures, Persistence, and the Line It Won't Cross

Know a Component Every Day: Redis

Author: 大大大大晴天

1. Introduction

Relational databases excel at persistence, transactions, complex queries, and data constraints, but in the following scenarios, directly hitting the database is often costly:

Many requests repeatedly read the same data;
Each request only performs a simple count or status check;
Leaderboards need frequent updates and sorting by score;
Temporary data like sessions and verification codes need automatic expiration;
Multiple application servers need to share short-term state;
Real-time events need to flow quickly between producers and consumers.

These operations may not involve large data volumes, but they can have very high access frequency. If every request falls through to a disk-backed database, it not only increases response time but also consumes database connections, CPU, and I/O.

Redis is a data structure server that uses memory as its primary data storage medium. It keeps the main data representation in memory and provides structured data operations over the network. Disk no longer sits on the critical path for every random read; it is primarily used for persistence, recovery, and backup. This trade-off brings low latency, but also means the dataset is limited by memory capacity, memory costs are higher, and durability needs separate design.

2. Redis Overall Architecture

A common single-instance Redis access path looks like this:

+-------------------+
| Application       |
| Java / Go / Python|
+---------+---------+
          |
          | Redis Protocol over TCP
          v
+------------------------------------------------+
| Redis Server                                   |
|                                                |
|  +---------------+     +--------------------+  |
|  | Network Event | --> | Command Execution  |  |
|  | Loop / I/O    |     | & Data Structures  |  |
|  +---------------+     +---------+----------+  |
|                                  |             |
|                    +-------------+-----------+ |
|                    |                         | |
|                    v                         v |
|              +-----------+             +-------+
|              | Keyspace  |             | Expire|
|              | in Memory |             | Evict |
|              +-----+-----+             +-------+
|                    |
|          +---------+----------+
|          |                    |
|          v                    v
|       RDB Snapshot         AOF Log
+----------+--------------------+-----------------+
           |                    |
           v                    v
        dump.rdb       appendonly files

2.1. Memory Is the Primary Workspace

Redis's data structures mainly reside in memory. When a command arrives, Redis can directly access in-memory objects without first reading data pages from disk. Compared to databases that use disk as the primary medium, this design reduces random I/O and the overhead of complex disk data structures.

However, "memory-based" does not mean "data never hits disk." Redis can persist its in-memory state to durable storage via RDB, AOF, or both.

2.2. Event-Driven and Non-Blocking I/O

After accepting a client connection, Redis sets the socket to non-blocking mode and creates a readable file event for it. A single Redis process can manage a large number of client connections with I/O multiplexing, without needing a business thread per connection.

A request roughly goes through this process:

Client sends command
      |
      v
Network layer reads request
      |
      v
Parse Redis protocol
      |
      v
Execute data structure operation
      |
      v
Generate response and write back to client

2.3. Why "Redis Is Single-Threaded" Isn't Accurate Enough

Many introductions attribute Redis's performance to being "single-threaded," which oversimplifies the real implementation.

A more accurate understanding is:

Redis's core command execution has long used a serialized model, thus avoiding a great deal of lock contention on data structures;
Network connections use event-driven and non-blocking I/O;
Persistence, asynchronous deletion, and some I/O work can use background processes or threads;
The specific threading model varies with Redis version and configuration.

2.4. Why Redis Is Fast

Redis's low latency comes from multiple factors, not a single trick:

Primary data lives in memory;
Data structures are designed for specific operations;
Core command execution reduces lock contention;
Non-blocking I/O and an event loop manage connections;
The Redis protocol is compact;
Pipelining reduces network round trips and system calls.

3. Redis Data Structures

Redis's core advantage is "server-side data structures," not just "key-value pairs in memory."

Data Type	Typical Commands	Use Cases	Main Caveats
String	SET, GET, INCR	Cache, counters, tokens	Large strings can become big keys
Hash	HSET, HGET	Object attributes, shopping carts	Too many fields: watch memory and iteration cost
List	LPUSH, RPOP, BLPOP	Simple queues, timelines	Not suitable for frequent random access
Set	SADD, SISMEMBER	Deduplication, tags, common relations	Large set operations can be slow
Sorted Set	ZADD, ZRANGE	Leaderboards, delayed tasks	Score is the sort key, not necessarily a unique business key
Stream	XADD, XREADGROUP	Event streams, consumer groups	Need to design ack, retry, and trimming
Bitmap	SETBIT, BITCOUNT	Check-ins, boolean status counts	Depends on stable integer offset mapping
HyperLogLog	PFADD, PFCOUNT	Approximate unique visitor counts	Result is an estimate, not exact
Geospatial	GEOADD, GEOSEARCH	Nearby location search	Not a replacement for full GIS systems

3.1. Express Business with Data Structures, Not as an In-Memory Map

The two approaches below both implement a leaderboard:

Option A:
user_id -> JSON(score)
Application reads all users, deserializes, sorts

Option B:
leaderboard -> Sorted Set
member = user_id
score  = business_score
Directly use ZADD / ZRANGE

Option B hands the sorting to Redis's sorted sets, which reduces application code and avoids moving all the data on every query.

This is also a key difference between Redis and a plain local cache: a local cache solves "how does this process quickly fetch a value," while Redis also solves "how do multiple processes share and atomically modify structured state."

3.2. The Boundary of Atomic Operations

A single Redis command usually executes in a way that no other command can interleave. For example, when multiple clients call INCR counter at the same time, updates are not lost due to application-layer "read-then-write" races.

When multiple commands need to be combined, you can consider:

MULTI / EXEC transactions;
WATCH for optimistic-lock-style conditional checks;
Lua scripts or Redis Functions;
Redesigning a complex operation into one native command.

Redis transactions guarantee that commands inside the transaction execute in order and that no other client's commands interleave during execution. However, this is not equivalent to a relational database transaction: Redis does not provide automatic rollback on runtime errors in the traditional sense. Even if some commands fail, other queued commands may still execute.

4. Expiration and Memory Eviction Are Not the Same Thing

Expiration: expiry time expresses the business lifecycle.

SET login:code:13800138000 731826 EX 300

This means the verification code becomes invalid after five minutes. Expiration is a business property of the key itself.

Eviction: eviction addresses what Redis should do with new writes when memory reaches maxmemory.

Memory has not reached maxmemory
        |
        v
     Normal write
        |
        v
Memory exceeds maxmemory
        |
        +--> noeviction: reject writes that may increase memory
        |
        +--> allkeys-*: select eviction candidates from all keys
        |
        +--> volatile-*: select from keys that have an expiration set

Redis supports LRU, LFU, random, and TTL policies. However, Redis implements approximate LRU/LFU, not the exact absolute order, which would be more costly to maintain.

If Redis only holds rebuildable cache, allkeys-lru or LFU based on access patterns can be considered. If Redis stores data that cannot be arbitrarily discarded, eviction policy should not be treated as a capacity management solution.

5. Persistence: How to Choose Between RDB and AOF

Redis offers four basic options:

RDB only;
AOF only;
Both RDB and AOF;
Persistence entirely disabled.

5.1. RDB: Saving a Snapshot at a Point in Time

RDB generates a snapshot of the dataset at a specific point in time.

In-memory data
   |
   | snapshot triggered periodically or manually
   v
 dump.rdb
   |
   | load on restart
   v
Restore in-memory state

Advantages:

The file is compact, easy to back up and transfer;
Relatively limited impact on the main process's normal request handling;
Large datasets usually load faster on restart.

Costs:

If a failure occurs between two snapshots, data written during that interval may be lost;
Creating a snapshot requires fork(). On large instances with tight memory or write-heavy loads, there can be additional memory and latency pressure.

5.2. AOF: Recording Write Commands

AOF appends commands that modify the dataset to a log, and replays those commands on restart to restore data.

SET / HSET / INCR ...
          |
          v
      AOF Buffer
          |
       fsync policy
          |
          v
   Append Only File

Common sync policies include:

appendfsync always: stronger durability, higher write overhead;
appendfsync everysec: usually a compromise between performance and durability;
appendfsync no: the OS decides when to flush, making the data loss window harder to control.

The default everysec policy may lose about one second of data under catastrophic failure. That "one second" is a design target, not an absolute upper bound across every hardware, filesystem, and failure model.

5.3. Combined Strategy

If Redis holds important data, you can use both RDB and AOF:

              +--> RDB: backup, faster recovery baseline
In-memory data -----|
              +--> AOF: narrow the loss window for recent writes

But even enabling both does not make Redis persistence a complete disaster recovery solution. At a minimum, consider:

Whether the persistence files are on reliable disks;
Whether there are off-machine or off-site backups;
Whether recovery is regularly rehearsed;
Whether there is enough memory during AOF rewrite and RDB snapshot;
How to recover if the host, availability zone, or entire cluster fails.

6. Differences Between Replication, Sentinel, and Cluster

6.1. Primary-Replica Replication

Redis replication uses a primary and replica model. The primary propagates data changes to replicas; if a connection is interrupted, the replica attempts partial synchronization, and falls back to full sync when partial sync is not possible.

                   write
Application -----------------> Primary
                                 |
                       async replication
                       +---------+---------+
                       |                   |
                       v                   v
                   Replica A           Replica B
                       ^                   ^
                       |                   |
                    read optional     read optional

Redis replication is asynchronous by default, so "the primary has acknowledged the write to the client" does not mean "the replica has necessarily durably saved that write." If the primary fails before replication completes, failover may lose recent writes.

Replication itself mainly provides data redundancy and read scaling; it does not alone deliver reliable automatic failover.

6.2. Sentinel

Sentinel targets non-Cluster deployments and provides:

Node monitoring;
Failure notification;
Automatic failover;
The current primary address to clients.

             +-------------+
             | Sentinel 1  |
             +------+------+
                    |
        +-----------+-----------+
        |                       |
+-------v-------+       +-------v-------+
|   Primary     | ----> |   Replica     |
+---------------+       +---------------+
        ^
        |
+-------+-------+
| Application   |
| Sentinel-aware|
+---------------+

Production environments typically run multiple Sentinel processes, and multiple Sentinels jointly determine failure. Running a single Sentinel makes Sentinel itself a single point of failure.

6.3. Redis Cluster

Cluster handles both data sharding and a degree of high availability.

Redis Cluster does not use classic consistent hashing; instead, it divides the keyspace into 16384 hash slots. Each primary node is responsible for a subset of these slots.

             CRC16(key) mod 16384
                       |
                       v
+--------------------------------------------------+
|                 16384 Hash Slots                 |
+----------------+----------------+----------------+
|   Node A       |    Node B      |    Node C      |
| slots 0..5500  | 5501..11000    | 11001..16383   |
+-------+--------+--------+-------+--------+--------+
        |                  |                 |
        v                  v                 v
    Replica A          Replica B         Replica C

Cluster clients need to handle redirects and maintain a mapping from slots to nodes. Multi-key commands usually require related keys to be in the same slot, which can be controlled with hash tags:

user:{1001}:profile
user:{1001}:orders
      ^----^
       same hash tag, so they go to the same slot

Cluster does not provide strong consistency. It uses asynchronous replication, and under certain failure windows or network partitions, acknowledged writes can still be lost; also, when a majority of primary nodes are unavailable, the cluster may stop serving.

6.4. How to Choose Between the Three Modes

Mode	Data Sharding	Automatic Failover	Typical Use Cases
Single instance	No	No	Local development, temporary cache
Primary-replica	No	No when used alone	Read scaling, data copies
Sentinel	No	Yes	HA when data fits on a single primary
Cluster	Yes	Yes	Data volume or throughput needs to scale across nodes

If a single node's capacity and throughput already suffice, Sentinel is often simpler than Cluster. Don't jump straight to Cluster just because "production" is in the sentence. Cluster introduces extra complexity in client compatibility, multi-key operations, slot migration, and failure diagnosis.

7. What Scenarios Suit Redis

7.1. Hot Cache

Client
  |
  v
Application ---- GET ----> Redis
  |                         |
  | cache miss              | cache hit
  v                         v
Database <---- query      Return
  |
  +---- SET EX ----> Redis

Suitable for caching product details, configuration, permission information, and computed results. Cached data should be reconstructable from an authoritative source.

The main benefit is not just speeding up a single read, but cutting repeated load on the backend database.

7.2. Counters and Rate Limiting

INCR, DECR, and expiration times work for:

Interface access counts;
Login failure counts;
Simple fixed-window rate limiting;
Simple counting in inventory pre-deduction.

Complex rate limiting usually requires Lua scripts or Functions to combine "read, check, update" into one atomic operation.

7.3. Sessions and Short-Term State

Redis is suited for storing:

Login sessions;
Verification codes;
OAuth temporary state;
Idempotency request markers;
Short-term lock state.

This data naturally needs sharing, low latency, and expiration.

7.4. Leaderboards

Sorted Sets can hold both members and scores, and query by score range or rank range, making them good for points boards, popularity boards, and live competition rankings.

7.5. Real-Time Event Streams

Pub/Sub is suitable for transient messages like online notifications where loss is acceptable. Redis Pub/Sub is at-most-once: once a message is sent, if a subscriber disconnects or fails to process it, Redis will not redeliver.

If you need message retention, consumer groups, acknowledgments, and pending-message tracking, Redis Streams should be the priority:

Producer --> Stream --> Consumer Group
                         |      |      |
                         v      v      v
                        C1     C2     C3
                         ______|______/
                          ACK / Pending

Even with Streams, the application still needs to design idempotency, retries, dead-letter processing, and stream trimming. It won't automatically handle full message governance for you.

7.6. Scenarios Where Redis Is Not a Good Fit

Be cautious in these scenarios:

Data size far exceeds what memory can afford;
Need for complex relational queries and ad-hoc analytics;
Need for strict cross-record ACID transactions;
Cannot accept the write loss window from asynchronous replication;
Want to use Pub/Sub for business messages that must be reliably delivered;
The team lacks the ability to maintain memory capacity, hot keys, and failover.

Redis can store critical data, but you must define the persistence, replication, and disaster recovery boundaries. It cannot automatically gain all the guarantees of a traditional database simply by enabling AOF.

8. Deployment and Usage

A simple deployment example:

docker run -d \
  --name redis-demo \
  -p 127.0.0.1:6379:6379 \
  redis:<version>

// connect to redis
docker exec -it redis-demo redis-cli

127.0.0.1:6379> PING
PONG

127.0.0.1:6379> SET article:1:title "Redis 入门"
OK

127.0.0.1:6379> GET article:1:title
"Redis 入门"

127.0.0.1:6379> INCR article:1:views
(integer) 1

127.0.0.1:6379> EXPIRE article:1:title 3600
(integer) 1

Redis selection decision tree:

                     Can the data be rebuilt?
                         /       \
                       yes        no
                      /            \
              Design as cache      Is brief data loss acceptable?
              TTL + eviction         /       \
                                     yes      no
                                   /            \
                            Persistence + replication      Re-evaluate tech stack
                            + DR rehearsal        or add authoritative storage
                                |
                        Does data exceed single-machine capacity?
                           /          \
                         no           yes
                        /                \
                 Sentinel optional    Cluster / sharding

Cache data: focus on hit rate, TTL, eviction policy, and database protection;
Temporary state: focus on expiration semantics, idempotency, and business behavior after failure;
Message data: focus on delivery semantics, acknowledgment, retries, and message retention;
Authoritative data: focus on persistence window, replication consistency, backup, and recovery;
Distributed coordination: focus on failure models, not just atomicity under the normal path.

9. Common Problems

9.1. Cache Penetration

Phenomenon: Requests query a key that doesn't exist in the database either, so every request bypasses the cache and hits the database.

Common measures:

Validate parameter legitimacy;
Cache empty results for a short time;
Use a Bloom Filter to filter out keys known not to exist;
Rate limit abnormal sources.

9.2. Cache Breakdown

Phenomenon: A high-frequency hot key expires, and a large number of requests hit the database simultaneously.

Common measures:

Mutex rebuild, allow only one request to load the data;
Logical expiration with background refresh;
More reasonable expiration policy for hot data;
Keep rate limiting and degradation on the database side.

9.3. Cache Avalanche

Phenomenon: A large number of keys expire at about the same time, or the entire Redis service is unavailable, causing traffic to flood back to the database.

Common measures:

Add reasonable random jitter to expiration times;
Pre-warm and refresh in batches;
Deploy highly available Redis;
Application-side rate limiting, circuit breaking, and degradation;
Reserve a controlled fallback capability for the database.

9.4. Big Keys

A big key could be:

An oversized string;
A collection with hundreds of thousands of members;
A hash with too many fields;
A stream that hasn't been trimmed for a long time.

It can cause slower network transfers, longer command execution, deletion latency, replication pressure, and slower failover.

Handling approaches:

Split at the data model layer;
Avoid returning all members at once;
Use asynchronous release methods like UNLINK where applicable;
Set trimming policies for streams;
Investigate with MEMORY USAGE, SCAN, and diagnostic tools.

9.5. Hot Keys

Hot keys are when access concentrates on a small number of keys. Even if memory is sufficient, a single key can bottleneck a Redis node, network link, or CPU core.

Common measures include:

Add a local cache layer in the application;
Shard splittable hot data;
Combine reads or use pipelining;
Rate limit hot endpoints;
Check for unnecessary duplicate reads.

9.6. Sudden Latency Spikes

Latency increase
   |
   +--> slow commands?        SLOWLOG
   |
   +--> big keys?             MEMORY USAGE / data sampling
   |
   +--> fork jitter?          RDB / AOF rewrite
   |
   +--> memory swapping?      OS metrics / swap
   |
   +--> network problems?     RTT / packet loss / connection pool
   |
   +--> client backlog?       CLIENT LIST / buffer
   |
   +--> CPU saturation?       host metrics