跪拜 Guibai
← Back to the summary

A Force-Push Disaster at 2 a.m. Produced a Git Workflow That Makes Rollback a Two-Command Job

Preface

🍊Reason

A force push disaster at 2 a.m.

Cover: The force push disaster at 2 a.m.

That night at 2 a.m., Xiao Li ran git push --force and wiped three days of commit history on master clean. No one knew which commit the production code corresponded to, no one dared to release, and the boss dropped a single word in the group: Investigate. During the post-mortem, the boss wrote up a team Git specification. Today it is published in full. As long as your team has more than three people, you can copy it and use it directly.

🏀Origin: Hello everyone, I am JavaDog程序狗. Today let's talk about something that can save your skin — a complete enterprise-level Git specification formulated by a team lead.

You have probably seen branch names like these:

branch-1
branch-2
branch-final
branch-real-final
branch-ultimate-edition-do-not-delete
zhangsan-branch
fix-something

The real trouble hits right before going live. Was this bug fixed or not? It was fixed, on some branch, but no one remembers which one. Which code is actually running in production? Can't tell — the version number was casually changed last Wednesday. Need to roll back? Roll back to where, how, and who takes responsibility?

To put it bluntly: you never nailed down where branches come from and where they merge to, so everyone flails around in the same pool.

The specification nails this down with a single table. Below I'll use the e-commerce v2.1.0 iteration as an example and walk through it from start to finish. After reading, you'll know how to avoid taking the blame next time.


🎯Main Objectives

This article walks you through a complete release cycle


Main Text

🍪Specification Breakdown

1. First, meet the two permanent employees

Branch What it is Who can modify
master The code currently running in production; every commit is directly deployable No one can modify directly; only via MR
develop The main development line; all features converge here Same as above

All other branches — whether feature, sprint, bugfix, release, or hotfix — are temporary workers. Delete them when the job is done.

👽Plain-language explanation master is the code running in production; every commit can go live directly. develop is the convergence point for all features. The remaining branches are all temporary workers — delete them once the work is done, don't keep them around.

This is the first line of defense. Code you casually commit directly to master can never appear. Want to touch master directly? No way — branch protection blocks you first.

Branch model: master and develop are permanent employees; others are temporary


2. The full story: E-commerce platform v2.1.0 iteration

First, some background: production is currently running v2.0.0. This iteration has two tasks — WeChat Pay upgrade assigned to Xiao Zhang, and user center refactoring assigned to Xiao Li. Yes, the same Xiao Li from the beginning. This time he behaved.

1. Iteration starts — create the sprint branch (done by the iteration lead)

An iteration needs a branch to collect the homework:

git checkout develop && git pull origin develop
git checkout -b sprint-v2.1
git push origin sprint-v2.1

Checkpoint: Don't touch the version number yet; it's still 2.0.0. Never change the version number during daily development. Remember this rule — it will be very useful later. So many people trip here: they get itchy and change the version number halfway through development, and in the end even they can't tell what the built artifact actually is.


2. Xiao Zhang develops WeChat Pay (done by the developer)

Iron rule: Never write code directly on sprint. Create your own feature branch. If you modify sprint directly, it's like the whole class sharing one piece of scratch paper — no one can tell who wrote what.

git checkout sprint-v2.1 && git pull origin sprint-v2.1
git checkout -b feature-wechat-pay-v2

Write code, commit:

git commit -m "feat(pay): add WeChat Pay v3 API support"

New code landed on sprint? Sync it — use rebase to keep a linear history:

git checkout sprint-v2.1 && git pull origin sprint-v2.1
git checkout feature-wechat-pay-v2
git rebase sprint-v2.1

Feature done? Don't merge it yourself. Create an MR: feature-wechat-pay-v2 into sprint-v2.1.

The MR will automatically run CI — compile, lint, unit tests, the full set. There's also an AI Code Review that scans first and gives suggestions — suggestions only, no blocking — so you don't have to wait for a colleague's review to find low-level mistakes. Then wait for one colleague to approve, and merge using Squash Merge.

Why Squash? Your commit history during development looks like this:

feat(pay): add WeChat Pay v3 API support
fix: fix something
fix: fix it again
fix: aaaah finally done

Makes your skin crawl just looking at it, right? Squash Merge compresses them into one clean commit. Sprint history stays clean from then on, and whoever picks it up later doesn't have to guess what those spasms in the middle were about.

After merging, the temp worker leaves:

git branch -d feature-wechat-pay-v2
git push origin --delete feature-wechat-pay-v2

Xiao Li's feature-user-center follows the same process.


3. Integration — merge sprint back into develop

Deploy sprint-v2.1 to the TEST environment (source compilation). Only after QA passes integration testing:

git checkout develop && git pull origin develop
git merge --no-ff sprint-v2.1     # Preserve iteration trace

git push origin develop

Why not Squash this time, but --no-ff? Remember one sentence:

Personal work branches get squashed (Squash); public milestones leave a trace (--no-ff).

--no-ff leaves an iteration summary node on the history graph. Ten years later you can still see at a glance exactly which features the v2.1 iteration contained. Squash and --no-ff are not an either-or choice — you squash personal work and leave traces for public milestones, each used in its own place.

There's an exception: when an iteration has only one feature, skip sprint and merge the feature directly into develop. Rules serve people, not the other way around.


4. Cut the release branch — only now do you set the version number

Pull release from develop, not from sprint:

git checkout develop && git pull origin develop
git checkout -b release-v2.1.0
git push origin release-v2.1.0

Now do two things:

  1. Change the version number: 2.0.0 becomes 2.1.0
  2. Replace SNAPSHOT dependencies with release versions

Iron rule: SNAPSHOT does not cross the wall. SNAPSHOT is a draft dependency that others can overwrite at any time. Build a package today using 1.2-SNAPSHOT, and tomorrow a colleague overwrites it — you can never reproduce the exact same package again. So SNAPSHOT is absolutely forbidden from entering master and PROD. The release branch is an isolation ward — all dependencies are converted to release versions here before publishing. No one gets to sneak a draft in halfway.


5. QA verification — build RC candidate images

docker build -t harbor.example.com/app:v2.1.0-rc1 .
docker push harbor.example.com/app:v2.1.0-rc1

QA's three-step process:

Step What to verify Example
① Main path regression High-frequency old features not broken Order, payment, refund still work
② Changed feature acceptance New features correct WeChat Pay works normally after upgrade
③ Performance observation No slowdown Payment API latency shows no significant degradation vs previous version

QA actually found a problem: the payment amount was off by 1 cent due to rounding. This bug going live means customer complaints; offline it means real money lost. What to do? Pull a bugfix from release and fix it:

git checkout -b bugfix-pay-rounding release-v2.1.0
# Fix the code...
git commit -m "fix(pay): correct rounding precision for amount"
# Squash Merge back into release (1 person Approve)
git checkout release-v2.1.0
git merge --squash bugfix-pay-rounding
git commit -m "fix(pay): correct rounding precision for amount"
git push origin release-v2.1.0
git branch -d bugfix-pay-rounding

After fixing, rebuild the image. Increment the rc number — it becomes v2.1.0-rc2 — and verify again until all green.

RC candidate versions can be overwritten when rebuilding, but release version Tags are never overwritten. This is the origin of the saying that a version has a single source of truth — when something goes wrong, you can point precisely to the commit and say, "It's this version."


6. Official release (master requires 2 Approvals)

git checkout master && git pull origin master
git merge --no-ff release-v2.1.0     # Requires 2 Approvals to merge
git tag -a v2.1.0 -m "Release v2.1.0"
git push origin master --tags

Build the official image; the release version gets the latest tag:

docker build -t harbor.example.com/app:v2.1.0 .
docker tag harbor.example.com/app:v2.1.0 harbor.example.com/app:latest
docker push harbor.example.com/app:v2.1.0
docker push harbor.example.com/app:latest

Blue-green deployment, simply put, means two identical servers. One (BLUE) is currently handling traffic. The new version is quietly deployed to the other (GREEN). After smoke testing confirms it's fine, flip the switch, traffic cuts over, and users feel nothing. This step is done by ops:

# 1. BLUE is currently online → deploy new version to GREEN
docker-compose -f docker-compose.green.yml up -d

# 2. Smoke test GREEN
curl http://green-host:8080/health

# 3. Cut traffic, both live
nginx -s reload

# 4. Sync upgrade BLUE to prepare for the next release
docker-compose -f docker-compose.blue.yml up -d

Blue-green deployment: dual-environment second-level switch

Don't forget the cleanup step: bugs fixed during the release period must be synced back to develop, then delete the release branch:

git checkout develop
git merge --no-ff release-v2.1.0
git push origin develop
git branch -d release-v2.1.0

v2.1.0 release complete.


7. The 2 a.m. incident: Hotfix and second-level rollback

The day after going live, users report occasional payment timeouts.

Scenario 1: Can locate, can fix — go the hotfix route

# Cut branch from master (not develop!)
git checkout master
git checkout -b hotfix-pay-timeout

# Fix, version PATCH +1: 2.1.0 → 2.1.1
git commit -m "fix(pay): increase payment gateway timeout to 10s"

# Merge back to master + Tag (2 Approvals)
git checkout master
git merge --no-ff hotfix-pay-timeout
git tag -a v2.1.1 -m "Hotfix: payment timeout"
git push origin master --tags

# Sync back to develop, delete branch
git checkout develop
git merge --no-ff hotfix-pay-timeout
git push origin develop
git branch -d hotfix-pay-timeout

Then build the v2.1.1 image and deploy via blue-green (same as step 6).

Scenario 2: Too severe, no time to fix — roll back directly

Two commands, second-level, zero downtime. The old version is still intact in the other environment:

sed -i 's/server app-green/server app-blue/' nginx/nginx.conf
nginx -s reload

Rollback: two commands for second-level recovery

After rolling back, you must immediately notify the entire team. Otherwise, developers will keep working on top of the broken code — you rolled back production, and they pushed it right back up.

Remember Xiao Li's force push from the beginning? Now master has branch protection, 2-person Approve, and non-overwritable Tags. Even if he wanted to force push, he'd have to pass two reviews first. That kind of accident is now physically impossible. Xiao Li has long since learned his lesson.


3. The whole process in one diagram

The whole process in one diagram


4. Five quick-reference tables (print-and-stick-on-wall edition)

Table 1: Where branches come from and where they go

Branch Pulled from Merged to One-liner
feature-* sprint sprint My feature, my rules
sprint-* develop develop One iteration, one collection point
bugfix-* sprint/release In place Dedicated to testing bugs
release-* develop master+develop Isolation ward before release
hotfix-* master master+develop Production fire brigade

Table 2: How to choose a merge strategy

Scenario Method Memory aid
feature/bugfix merged in Squash Squash personal branches
sprint/release/hotfix merged in --no-ff Leave a trace for milestones
Syncing the latest code Rebase Move house, keep it linear

Table 3: When to change the version number

Timing Change Example
When cutting release Set this version 2.0.0 → 2.1.0
When cutting hotfix PATCH +1 2.1.0 → 2.1.1
Any other time Don't touch

Table 4: Commit Message cheat sheet

feat(pay): add WeChat Pay v3 API support
fix(pay): correct rounding precision for amount
fix(pay): increase payment gateway timeout to 10s
docs(readme): update deployment guide
refactor(user): extract login service from controller

Rules: type(scope) starts with a verb in lowercase English, no more than 72 characters, no period at the end; BREAKING CHANGE goes in all caps in the footer. The supporting tool trio — commitlint, husky, standard-version — makes format issues disappear forever. No one will ever argue with you about how to write a commit message again.

Table 5: Environments and approval thresholds

Environment Deployment method Source Approval
TEST Source compilation sprint/release/develop 1 Approve
PROD Docker image (manual trigger) Only master release images 2 Approvals

When TEST resources conflict, priority is hotfix > release > develop > sprint. Production incidents always come first — this is non-negotiable.


5. Five red lines — don't cross a single one

  1. git push --force to a public branch. This is the root of the opening accident — it directly destroys others' history. If someone crosses this line, I want to revoke their branch permissions on the spot.
  2. Committing directly on master/develop. All protections are bypassed — it's like leaving the door unlocked.
  3. Merging without Code Review. Quality is running naked; no one can save you if something goes wrong.
  4. SNAPSHOT dependencies entering master. Impossible to reproduce after going live. When you get called at midnight, you won't even be able to find the cause through your tears.
  5. Not notifying the team after a rollback. This is a collaboration disaster — others think production is fine and keep piling on code.

🍈You Might Want to Ask

How to contact Dog Brother for discussion?

Follow the public account 【JavaDog程序狗】, reply 【入群】 or 【加入】 to chat about tech and share war stories.