A Force-Push Disaster at 2 a.m. Produced a Git Workflow That Makes Rollback a Two-Command Job
Preface
🍊Reason
A force push disaster at 2 a.m.
That night at 2 a.m., Xiao Li ran
git push --forceand wiped three days of commit history on master clean. No one knew which commit the production code corresponded to, no one dared to release, and the boss dropped a single word in the group: Investigate. During the post-mortem, the boss wrote up a team Git specification. Today it is published in full. As long as your team has more than three people, you can copy it and use it directly.
🏀Origin: Hello everyone, I am JavaDog程序狗. Today let's talk about something that can save your skin — a complete enterprise-level Git specification formulated by a team lead.
You have probably seen branch names like these:
branch-1
branch-2
branch-final
branch-real-final
branch-ultimate-edition-do-not-delete
zhangsan-branch
fix-something
The real trouble hits right before going live. Was this bug fixed or not? It was fixed, on some branch, but no one remembers which one. Which code is actually running in production? Can't tell — the version number was casually changed last Wednesday. Need to roll back? Roll back to where, how, and who takes responsibility?
To put it bluntly: you never nailed down where branches come from and where they merge to, so everyone flails around in the same pool.
The specification nails this down with a single table. Below I'll use the e-commerce v2.1.0 iteration as an example and walk through it from start to finish. After reading, you'll know how to avoid taking the blame next time.
🎯Main Objectives
This article walks you through a complete release cycle
- Meet the two permanent employees: master and develop
- From branching to going live, 7 steps without stepping on landmines
- Blue-green deployment and second-level rollback — don't panic when incidents strike
- Five quick-reference tables + five red lines — print them, stick them on the wall, and check anytime
Main Text
🍪Specification Breakdown
1. First, meet the two permanent employees
| Branch | What it is | Who can modify |
|---|---|---|
master |
The code currently running in production; every commit is directly deployable | No one can modify directly; only via MR |
develop |
The main development line; all features converge here | Same as above |
All other branches — whether feature, sprint, bugfix, release, or hotfix — are temporary workers. Delete them when the job is done.
👽Plain-language explanation master is the code running in production; every commit can go live directly. develop is the convergence point for all features. The remaining branches are all temporary workers — delete them once the work is done, don't keep them around.
This is the first line of defense. Code you casually commit directly to master can never appear. Want to touch master directly? No way — branch protection blocks you first.
2. The full story: E-commerce platform v2.1.0 iteration
First, some background: production is currently running v2.0.0. This iteration has two tasks — WeChat Pay upgrade assigned to Xiao Zhang, and user center refactoring assigned to Xiao Li. Yes, the same Xiao Li from the beginning. This time he behaved.
1. Iteration starts — create the sprint branch (done by the iteration lead)
An iteration needs a branch to collect the homework:
git checkout develop && git pull origin develop
git checkout -b sprint-v2.1
git push origin sprint-v2.1
Checkpoint: Don't touch the version number yet; it's still 2.0.0. Never change the version number during daily development. Remember this rule — it will be very useful later. So many people trip here: they get itchy and change the version number halfway through development, and in the end even they can't tell what the built artifact actually is.
2. Xiao Zhang develops WeChat Pay (done by the developer)
Iron rule: Never write code directly on sprint. Create your own feature branch. If you modify sprint directly, it's like the whole class sharing one piece of scratch paper — no one can tell who wrote what.
git checkout sprint-v2.1 && git pull origin sprint-v2.1
git checkout -b feature-wechat-pay-v2
Write code, commit:
git commit -m "feat(pay): add WeChat Pay v3 API support"
New code landed on sprint? Sync it — use rebase to keep a linear history:
git checkout sprint-v2.1 && git pull origin sprint-v2.1
git checkout feature-wechat-pay-v2
git rebase sprint-v2.1
Feature done? Don't merge it yourself. Create an MR: feature-wechat-pay-v2 into sprint-v2.1.
The MR will automatically run CI — compile, lint, unit tests, the full set. There's also an AI Code Review that scans first and gives suggestions — suggestions only, no blocking — so you don't have to wait for a colleague's review to find low-level mistakes. Then wait for one colleague to approve, and merge using Squash Merge.
Why Squash? Your commit history during development looks like this:
feat(pay): add WeChat Pay v3 API support
fix: fix something
fix: fix it again
fix: aaaah finally done
Makes your skin crawl just looking at it, right? Squash Merge compresses them into one clean commit. Sprint history stays clean from then on, and whoever picks it up later doesn't have to guess what those spasms in the middle were about.
After merging, the temp worker leaves:
git branch -d feature-wechat-pay-v2
git push origin --delete feature-wechat-pay-v2
Xiao Li's feature-user-center follows the same process.
3. Integration — merge sprint back into develop
Deploy sprint-v2.1 to the TEST environment (source compilation). Only after QA passes integration testing:
git checkout develop && git pull origin develop
git merge --no-ff sprint-v2.1 # Preserve iteration trace
git push origin develop
Why not Squash this time, but --no-ff? Remember one sentence:
Personal work branches get squashed (Squash); public milestones leave a trace (--no-ff).
--no-ff leaves an iteration summary node on the history graph. Ten years later you can still see at a glance exactly which features the v2.1 iteration contained. Squash and --no-ff are not an either-or choice — you squash personal work and leave traces for public milestones, each used in its own place.
There's an exception: when an iteration has only one feature, skip sprint and merge the feature directly into develop. Rules serve people, not the other way around.
4. Cut the release branch — only now do you set the version number
Pull release from develop, not from sprint:
git checkout develop && git pull origin develop
git checkout -b release-v2.1.0
git push origin release-v2.1.0
Now do two things:
- Change the version number: 2.0.0 becomes 2.1.0
- Replace SNAPSHOT dependencies with release versions
Iron rule: SNAPSHOT does not cross the wall. SNAPSHOT is a draft dependency that others can overwrite at any time. Build a package today using 1.2-SNAPSHOT, and tomorrow a colleague overwrites it — you can never reproduce the exact same package again. So SNAPSHOT is absolutely forbidden from entering master and PROD. The release branch is an isolation ward — all dependencies are converted to release versions here before publishing. No one gets to sneak a draft in halfway.
5. QA verification — build RC candidate images
docker build -t harbor.example.com/app:v2.1.0-rc1 .
docker push harbor.example.com/app:v2.1.0-rc1
QA's three-step process:
| Step | What to verify | Example |
|---|---|---|
| ① Main path regression | High-frequency old features not broken | Order, payment, refund still work |
| ② Changed feature acceptance | New features correct | WeChat Pay works normally after upgrade |
| ③ Performance observation | No slowdown | Payment API latency shows no significant degradation vs previous version |
QA actually found a problem: the payment amount was off by 1 cent due to rounding. This bug going live means customer complaints; offline it means real money lost. What to do? Pull a bugfix from release and fix it:
git checkout -b bugfix-pay-rounding release-v2.1.0
# Fix the code...
git commit -m "fix(pay): correct rounding precision for amount"
# Squash Merge back into release (1 person Approve)
git checkout release-v2.1.0
git merge --squash bugfix-pay-rounding
git commit -m "fix(pay): correct rounding precision for amount"
git push origin release-v2.1.0
git branch -d bugfix-pay-rounding
After fixing, rebuild the image. Increment the rc number — it becomes v2.1.0-rc2 — and verify again until all green.
RC candidate versions can be overwritten when rebuilding, but release version Tags are never overwritten. This is the origin of the saying that a version has a single source of truth — when something goes wrong, you can point precisely to the commit and say, "It's this version."
6. Official release (master requires 2 Approvals)
git checkout master && git pull origin master
git merge --no-ff release-v2.1.0 # Requires 2 Approvals to merge
git tag -a v2.1.0 -m "Release v2.1.0"
git push origin master --tags
Build the official image; the release version gets the latest tag:
docker build -t harbor.example.com/app:v2.1.0 .
docker tag harbor.example.com/app:v2.1.0 harbor.example.com/app:latest
docker push harbor.example.com/app:v2.1.0
docker push harbor.example.com/app:latest
Blue-green deployment, simply put, means two identical servers. One (BLUE) is currently handling traffic. The new version is quietly deployed to the other (GREEN). After smoke testing confirms it's fine, flip the switch, traffic cuts over, and users feel nothing. This step is done by ops:
# 1. BLUE is currently online → deploy new version to GREEN
docker-compose -f docker-compose.green.yml up -d
# 2. Smoke test GREEN
curl http://green-host:8080/health
# 3. Cut traffic, both live
nginx -s reload
# 4. Sync upgrade BLUE to prepare for the next release
docker-compose -f docker-compose.blue.yml up -d
Don't forget the cleanup step: bugs fixed during the release period must be synced back to develop, then delete the release branch:
git checkout develop
git merge --no-ff release-v2.1.0
git push origin develop
git branch -d release-v2.1.0
v2.1.0 release complete.
7. The 2 a.m. incident: Hotfix and second-level rollback
The day after going live, users report occasional payment timeouts.
Scenario 1: Can locate, can fix — go the hotfix route
# Cut branch from master (not develop!)
git checkout master
git checkout -b hotfix-pay-timeout
# Fix, version PATCH +1: 2.1.0 → 2.1.1
git commit -m "fix(pay): increase payment gateway timeout to 10s"
# Merge back to master + Tag (2 Approvals)
git checkout master
git merge --no-ff hotfix-pay-timeout
git tag -a v2.1.1 -m "Hotfix: payment timeout"
git push origin master --tags
# Sync back to develop, delete branch
git checkout develop
git merge --no-ff hotfix-pay-timeout
git push origin develop
git branch -d hotfix-pay-timeout
Then build the v2.1.1 image and deploy via blue-green (same as step 6).
Scenario 2: Too severe, no time to fix — roll back directly
Two commands, second-level, zero downtime. The old version is still intact in the other environment:
sed -i 's/server app-green/server app-blue/' nginx/nginx.conf
nginx -s reload
After rolling back, you must immediately notify the entire team. Otherwise, developers will keep working on top of the broken code — you rolled back production, and they pushed it right back up.
Remember Xiao Li's force push from the beginning? Now master has branch protection, 2-person Approve, and non-overwritable Tags. Even if he wanted to force push, he'd have to pass two reviews first. That kind of accident is now physically impossible. Xiao Li has long since learned his lesson.
3. The whole process in one diagram
4. Five quick-reference tables (print-and-stick-on-wall edition)
Table 1: Where branches come from and where they go
| Branch | Pulled from | Merged to | One-liner |
|---|---|---|---|
| feature-* | sprint | sprint | My feature, my rules |
| sprint-* | develop | develop | One iteration, one collection point |
| bugfix-* | sprint/release | In place | Dedicated to testing bugs |
| release-* | develop | master+develop | Isolation ward before release |
| hotfix-* | master | master+develop | Production fire brigade |
Table 2: How to choose a merge strategy
| Scenario | Method | Memory aid |
|---|---|---|
| feature/bugfix merged in | Squash | Squash personal branches |
| sprint/release/hotfix merged in | --no-ff | Leave a trace for milestones |
| Syncing the latest code | Rebase | Move house, keep it linear |
Table 3: When to change the version number
| Timing | Change | Example |
|---|---|---|
| When cutting release | Set this version | 2.0.0 → 2.1.0 |
| When cutting hotfix | PATCH +1 | 2.1.0 → 2.1.1 |
| Any other time | Don't touch | — |
Table 4: Commit Message cheat sheet
feat(pay): add WeChat Pay v3 API support
fix(pay): correct rounding precision for amount
fix(pay): increase payment gateway timeout to 10s
docs(readme): update deployment guide
refactor(user): extract login service from controller
Rules: type(scope) starts with a verb in lowercase English, no more than 72 characters, no period at the end; BREAKING CHANGE goes in all caps in the footer. The supporting tool trio — commitlint, husky, standard-version — makes format issues disappear forever. No one will ever argue with you about how to write a commit message again.
Table 5: Environments and approval thresholds
| Environment | Deployment method | Source | Approval |
|---|---|---|---|
| TEST | Source compilation | sprint/release/develop | 1 Approve |
| PROD | Docker image (manual trigger) | Only master release images | 2 Approvals |
When TEST resources conflict, priority is hotfix > release > develop > sprint. Production incidents always come first — this is non-negotiable.
5. Five red lines — don't cross a single one
git push --forceto a public branch. This is the root of the opening accident — it directly destroys others' history. If someone crosses this line, I want to revoke their branch permissions on the spot.- Committing directly on master/develop. All protections are bypassed — it's like leaving the door unlocked.
- Merging without Code Review. Quality is running naked; no one can save you if something goes wrong.
- SNAPSHOT dependencies entering master. Impossible to reproduce after going live. When you get called at midnight, you won't even be able to find the cause through your tears.
- Not notifying the team after a rollback. This is a collaboration disaster — others think production is fine and keep piling on code.
🍈You Might Want to Ask
How to contact Dog Brother for discussion?
Follow the public account 【JavaDog程序狗】, reply 【入群】 or 【加入】 to chat about tech and share war stories.