A Production Incident: Added Redisson Distributed Lock, but Data Still Got Corrupted by Concurrency

A while back, I took on a requirement with a very simple scenario: users concurrently operating on a certain resource (such as claiming limited coupons, or frequently modifying a configuration). For this kind of anti-concurrency overselling scenario, any veteran Java developer can think of a combo with their eyes closed: Spring Boot + @Transactional + Redisson distributed lock.

I banged out the code smoothly, tested it locally with a single thread without issues, and directly pushed it to testing and then production. On the first day after going live, operations came over and said: "The backend data is wrong, why was the same resource deducted twice?"

I was shocked and quickly checked the logs. The more I looked, the more numb I felt: Although a distributed lock was added, under extremely high concurrency, the lock actually "failed"!

Today, let me review this pitfall that almost got me a performance rating of C.

Recreating the "Suicidal" Code at That Time

To make it intuitive, I simplified the business code from back then. The general logic was: query the remaining amount in the database -> check if it is sufficient -> deduct -> save.

Java

@Service
public class ResourceServiceImpl implements ResourceService {

    @Autowired
    private RedissonClient redissonClient;
    @Autowired
    private ResourceMapper resourceMapper;

    @Override
    @Transactional(rollbackFor = Exception.class)
    public void consumeResource(String resourceId) {
        String lockKey = "lock:resource:" + resourceId;
        RLock lock = redissonClient.getLock(lockKey);

        try {
            // Try to acquire the lock, wait up to 3 seconds
            if (lock.tryLock(3, TimeUnit.SECONDS)) {
                // 1. Query current stock from database
                Resource res = resourceMapper.selectById(resourceId);
                if (res.getStock() > 0) {
                    // 2. Deduct
                    res.setStock(res.getStock() - 1);
                    resourceMapper.updateById(res);
                } else {
                    throw new RuntimeException("Insufficient stock");
                }
            } else {
                throw new RuntimeException("System busy, please retry");
            }
        } catch (InterruptedException e) {
            Thread.currentThread().interrupt();
        } finally {
            // 3. Release lock
            if (lock.isLocked() && lock.isHeldByCurrentThread()) {
                lock.unlock();
            }
        }
    }
}

Take a look, everyone, doesn't this code look familiar? @Transactional ensures the transaction, try...finally ensures the lock is definitely released. No matter how you review this code, the logic seems flawless!

But once concurrency hit, the database inventory still went negative.

Finding the Culprit: The Time Gap Between Lock Release and Transaction Commit

After investigating for most of the day, comparing MySQL's binlog and Redis execution logs, I finally found the problem: It's Spring AOP's fault.

Recall the underlying principle of @Transactional. Spring implements transactions through AOP dynamic proxies, which is equivalent to wrapping a layer around your business method:

Java

// Pseudocode of Spring proxy class
public void proxyConsumeResource(String resourceId) {
    // 1. Start database transaction
    connection.setAutoCommit(false); 
    try {
        // 2. Execute your real business logic (including locking, modifying data, releasing lock)
        target.consumeResource(resourceId); 
        // 3. Commit transaction
        connection.commit(); 
    } catch (Exception e) {
        connection.rollback();
    }
}

Do you see the fatal problem?!

In my business code, the finally block executed lock.unlock(), at which point the distributed lock had already been released. However! At this moment, the target.consumeResource() method had just finished executing, but the Spring proxy class's connection.commit() had not yet executed!

That is to say, the lock was gone, but the data had not yet been persisted to disk.

At this moment, if another thread (Thread B) comes in concurrently:

1. Thread B sees that there is no lock in Redis and successfully acquires the lock.
2. Thread B queries the database. Because Thread A's transaction has not yet committed, Thread B still reads the old data!
3. Thread B performs the deduction based on the old data.
4. Thread A commits the transaction, and Thread B commits its transaction immediately after.

Perfect, a dirty write occurred, and the data was completely corrupted.

Fixing the Pitfall: Let the Lock Fly a Little Longer

After finding the cause, the solution is very simple. The core idea is only one: You must ensure the lock is released after the transaction commits.

Solution 1: Brute-force Split (Recommended)

Directly move the locking logic up to the controller layer, or wrap it in another service layer. Ensure the scope of the lock is larger than the scope of the transaction.

Java

@Service
public class ResourceLockService {

    @Autowired
    private RedissonClient redissonClient;
    @Autowired
    private ResourceServiceImpl resourceService; // Inject the original transactional Service

    public void safeConsume(String resourceId) {
        String lockKey = "lock:resource:" + resourceId;
        RLock lock = redissonClient.getLock(lockKey);

        try {
            if (lock.tryLock(3, TimeUnit.SECONDS)) {
                // Call the transactional method. Since the transactional method is an independent proxy,
                // when execution returns here, the transaction has already committed!
                resourceService.consumeResource(resourceId); 
            }
        } catch (InterruptedException e) {
            Thread.currentThread().interrupt();
        } finally {
            if (lock.isLocked() && lock.isHeldByCurrentThread()) {
                lock.unlock();
            }
        }
    }
}

Note: Do not write these two methods in the same class and call them directly via this, as this will cause AOP to fail—a well-known issue.

Solution 2: Programmatic Transaction

If you don't want to write an extra layer of class, you can use TransactionTemplate to manually control the transaction boundary.

Java

// Execute after acquiring the lock:
transactionTemplate.execute(status -> {
    // Query database, deduct, update
    return null;
});
// After the transaction commits, enter finally to release the lock

A Pitfall Worth Mentioning: Redisson Watchdog Failure

Taking this opportunity, let me share another pitfall many people easily fall into when using Redisson. Some folks like to pass a leaseTime (lock expiration time) when locking:

Java

// Attempt to lock, wait 3 seconds, automatically release after locking for 10 seconds
lock.tryLock(3, 10, TimeUnit.SECONDS);

Once you explicitly pass in leaseTime, Redisson's WatchDog mechanism will be disabled! If your business logic execution time exceeds 10 seconds (for example, due to a Full GC or calling a very slow third-party interface), the lock will be automatically released, and other threads will sneak in.

The correct approach is, if you don't know exactly how long the business will execute, never pass leaseTime:

Java

// Only pass wait time, no leaseTime, watchdog mechanism is active and will automatically renew for you
lock.tryLock(3, TimeUnit.SECONDS);

Summary

In daily business development, when using @Transactional together with various locks (including local locks like synchronized and distributed locks), you must pay extra attention and map out their scope boundaries.

Sometimes it's not that the underlying components are bad, but purely that we haven't figured out the execution order of Spring AOP. I hope my painful lesson can help you all lose a few less hairs. What outrageous concurrency pitfalls have you encountered in production? Welcome to share in the comments~