The Five Backend Requirements That Sound Simple but Aren't

When the Boss Says "Just Add a Field to the Database" — A Backend Developer's Collection of Bizarre Requirements

Topic: #BizarreRequirementsCollection

Foreword

Hello everyone, I'm DreamCat, a full-stack independent developer. I've previously discussed the AI era and bizarre requirements from a frontend perspective. Today, let's switch gears — and talk about those blood-pressure-spiking "simple requirements" in backend development.

After doing backend work for a long time, you'll notice a pattern: The shorter the requirement, the bigger the pitfall.

"Just add a field to the database" — actually requires modifying 5 tables, 3 APIs, and 2 scheduled tasks.
"Add an export function, just pull an Excel file" — data volume is 2 million rows, and you get an OOM error.
"Add a scheduled task, run it once a day" — then you get woken up by an alert call at 3 AM.

These requirements might take a product manager only 10 seconds to state, but implementing them often means countless late nights and moments of self-doubt asking "why was it designed this way."

Today, let's talk about a few bizarre backend development requirements I've personally experienced, and how I survived them.

Requirement 1: "Just Add a Field, Isn't a Database Just Adding a Column?"

Background

At the end of last year, I took over the maintenance of an e-commerce system. One day, a product manager suddenly approached me: "A small requirement — add a field to the product table to record a product recommendation score, making it easier for operations to sort."

I thought to myself: Hmm, indeed a small requirement.

Then he added: "Oh, right, this recommendation score should automatically update every day at midnight, calculated based on sales volume, ratings, favorites count, page views, and return rate. Also, the operations backend needs to be able to manually adjust it, with a history log. And this field needs access control based on permissions."

I fell silent.

Technical Analysis

The actual workload of this requirement was worlds apart from "adding a field":

Surface requirement: Add a field
Actual requirement:
├── Database: Add a field to the product table, but also create a history log table
├── Scheduled task: Calculate the recommendation score every day at midnight
├── Algorithm: A weighted formula for sales, ratings, favorites, views, and return rate
├── API: Manual adjustment in the operations backend + permission control + change log
├── Cache: The product list page reads this field, requiring a caching strategy
└── Compatibility: Database migration plan + rollback plan

The biggest pitfall was that this "simple requirement" hid a distributed computing challenge.

We had 2 million products at the time. If we calculated all of them at midnight, even at 10ms per item, it would take 20,000 seconds — nearly 6 hours. By the time it finished, the sun would be up.

My Solution

I spent 3 days designing the following:

Incremental updates: Only calculate products that changed that day (sales changed, new reviews, etc.), rather than a full recalculation.
Sharded parallelism: Use a message queue to distribute product IDs across 10 workers for parallel processing.
Fallback full recalculation: Run a complete recalculation every Sunday to prevent accumulated errors from incremental updates.

# Incremental update plan for recommendation scores
import redis
from celery import Celery
from typing import List, Dict

app = Celery('recommend', broker='redis://localhost:6379/0')

# Weight configuration (dynamically adjustable)
WEIGHTS = {
    'sales': 0.35,
    'rating': 0.25,
    'favorites': 0.15,
    'views': 0.15,
    'return_rate': -0.10  # Return rate is a negative indicator
}

# Mark which products need updating
def mark_product_dirty(product_id: str):
    """When a product changes, mark it for recalculation"""
    redis.sadd('recommend:dirty_products', product_id)

@app.task
def update_recommend_scores():
    """Midnight scheduled task: incrementally update recommendation scores"""
    # 1. Get the IDs of products that need updating
    dirty_ids = redis.smembers('recommend:dirty_products')
    if not dirty_ids:
        return

    # 2. Shard (batches of 1000)
    batch_size = 1000
    batches = [list(dirty_ids)[i:i+batch_size]
               for i in range(0, len(dirty_ids), batch_size)]

    # 3. Parallel processing
    for batch in batches:
        update_batch.delay(batch)

    # 4. Clear dirty flags
    redis.delete('recommend:dirty_products')

@app.task
def update_batch(product_ids: List[str]):
    """Batch update recommendation scores"""
    scores = {}
    for pid in product_ids:
        # Get metric data from cache/database
        metrics = get_product_metrics(pid)
        # Weighted calculation
        score = sum(
            metrics.get(k, 0) * w
            for k, w in WEIGHTS.items()
        )
        scores[pid] = round(score, 2)

    # Batch write to database
    batch_update_product_scores(scores)

Results

After going live, the recommendation score calculation time dropped from 6 hours to 15 minutes. The sorting feature in the operations backend also launched smoothly.

But what made me laugh and cry was — the product manager later told me: "Actually, you didn't need to make it so complicated. I just wanted to add a field, and operations could fill it in manually."

Me: ...Then what about all those requirements you mentioned before?

Him: "Oh, those were just things I thought we might as well do since we were already working on it."

This requirement taught me one thing: The product manager's "by the way" and "since we're at it" are the biggest pitfalls in backend development.

Requirement 2: "The Export Function is Simple, Just Pull an Excel File"

Background

In another project, a client needed a data export function. His exact words were: "Just that data table in the backend — add an export button, click to download an Excel file. Very simple, right?"

I asked him: "What's the approximate data volume?"

He said: "Not much, just tens of thousands of rows."

I checked the database — good lord, 2 million rows.

Technical Analysis

The pitfall of an export function isn't in the implementation itself, but in the data volume.

Data Volume	Solution	Risk
< 10k	One-time query + file generation	Minimal risk
10k - 100k	Paginated query + streaming write	Watch memory
100k - 1M	Async export + sharding + queue	Requires a task system
> 1M	Async export + multiple workers + file merging	Quite complex

Exporting 2 million rows at once = memory explosion = server OOM = getting woken up in the middle of the night.

My Solution

# Async export solution
import asyncio
from io import BytesIO
import openpyxl
from openpyxl.styles import Font, Alignment, PatternFill

class AsyncExcelExporter:
    """Async Excel exporter"""
    def __init__(self, export_id: str, query_params: dict):
        self.export_id = export_id
        self.query_params = query_params
        self.batch_size = 5000  # 5000 rows per batch

    async def export(self):
        # 1. First, query the total count
        total = await self.get_total_count()
        # 2. Update the export task status
        await self.update_progress(0, total)

        # 3. Create the Excel file
        wb = openpyxl.Workbook()
        ws = wb.active
        # Write headers
        headers = ['ID', 'Product Name', 'Price', 'Sales', 'Rating', 'Created At']
        ws.append(headers)

        # 4. Paginated query + streaming write
        offset = 0
        while offset < total:
            batch = await self.query_batch(offset, self.batch_size)
            for row in batch:
                ws.append(row)
            offset += self.batch_size
            # Update progress
            await self.update_progress(min(offset, total), total)
            # Release memory every 50k rows
            if offset % 50000 == 0:
                gc.collect()

        # 5. Save to file/OSS
        buffer = BytesIO()
        wb.save(buffer)
        buffer.seek(0)
        file_url = await self.upload_to_oss(buffer)

        # 6. Notify the user to download
        await self.notify_user(file_url)
        return file_url

Results

After the async export went live, clicking the "Export" button would show a message: "Export task created. You will be notified when it's ready for download." Once the export was complete, the system automatically sent a notification, and the user could click the link to download.

Client feedback: "This export function is well done, but I later realized I actually don't need to export all the data. Generally, exporting the filtered few thousand rows is enough."

I fell silent again.

Requirement 3: "Don't Worry About Security, Just Get It Online First"

Background

This requirement came from a startup team. They were in a rush to launch their MVP, and the boss said: "Don't worry about security for now. Let's just get the features running first, and we'll patch it up later."

I built an internal admin dashboard, but the boss said "logging in is too troublesome, can we remove it?" So I removed the login and launched it completely exposed.

Then — two weeks later, user data was scraped.

Technical Analysis

"Launch first, patch security later" is the most common pitfall in startup projects. Security is not a "feature" — it's infrastructure. The cost of patching security afterward is 10 times higher than doing it upfront.

Common "launch first" security traps:

1. No authentication at all → Exposed APIs, data scraped
2. Passwords stored in plain text → Database leak = user password leak
3. SQL concatenation → SQL injection, database deletion
4. File uploads without validation → Server compromised with malware
5. No rate limiting on APIs → APIs abused, bill explodes
6. Sensitive info printed in logs → Passwords, tokens all over the logs

My Lesson

After this incident, I created a "Minimum Security Checklist" for myself:

# No matter how rushed you are, these must be done:
MINIMUM_SECURITY_CHECKLIST = [
    '✅ At least JWT authentication on all APIs',
    '✅ Password hashing (bcrypt/argon2), absolutely no plain text',
    '✅ Use ORM parameterized queries, prohibit SQL concatenation',
    '✅ File uploads: restrict type, size, scan for viruses',
    '✅ Rate limiting on sensitive APIs',
    '✅ Log sanitization in production (do not print passwords, tokens)',
    '✅ CORS whitelist configuration',
    '✅ Enforce HTTPS',
    '✅ Database access IP whitelist',
]

Similar incidents are hard to prevent entirely. I later gave the team a lesson:

Security is not a "feature"; it's the foundation. If the foundation isn't solid, no matter how high you build, it will collapse. And when you fix the foundation, everyone in the building has to move out — can you afford that cost?

Requirement 4: "Switch the Database from MySQL to MongoDB, Because I Heard It's Faster"

Background

This requirement came from a CTO. He attended a MongoDB talk at a tech conference and came back deciding: "Our user system's data volume is too large, MySQL can't handle it. Let's switch to MongoDB."

I asked him: "What query scenarios does the user system have?"

He said: "Login, registration, querying user info, editing profiles, paginated lists... just those."

I said: "MySQL can handle all these scenarios perfectly, and we're using relational data."

He said: "But MongoDB is faster. NoSQL is the trend."

Technical Analysis

The core issue with this requirement isn't about technology choice, but that "I heard it's faster" is the most dangerous reason for choosing a technology.

Core differences between MySQL and MongoDB:

Dimension	MySQL	MongoDB
Data Model	Relational, tables + rows	Document, collections + documents
Use Case	Structured data, transactions	Semi-structured data, flexible schema
Join Queries	Strong JOIN capability	Not suitable for multi-table joins
Transactions	ACID, mature	Multi-document transactions supported since 4.0+
Data Consistency	Strong consistency	Eventually consistent by default

A user system is a typical relational data scenario:

Users table ←→ User profiles table (1:1)
Users table ←→ Orders table (1:N)
Users table ←→ Shipping addresses table (1:N)
Users table ←→ Follow relationships table (N:N, self-referencing)

Switching this scenario to MongoDB is like using a hammer to turn a screw — it's not that you can't, but you'll regret not using a screwdriver.

My Solution

I didn't refuse outright. Instead, I did three things:

Wrote a POC using MongoDB: Implemented the core user system features with MongoDB.
Comparative testing: Compared query performance between the two solutions with the same data volume.
Listed the real problems: MySQL wasn't slow because of the database itself, but because of missing indexes and unoptimized queries.

-- Test results: After adding indexes, MySQL's performance was more than sufficient
-- Before optimization: Full table scan, 200ms
SELECT * FROM users WHERE email = '[email protected]';

-- After optimization: Added index, 0.5ms
CREATE INDEX idx_users_email ON users(email);
SELECT * FROM users WHERE email = '[email protected]';

Conclusion: MySQL wasn't the problem; our indexes were wrong.

Results

In the end, the CTO saw the comparison data and abandoned the migration. We spent 3 days optimizing indexes and queries, and the user system's response time dropped from 200ms to 5ms.

Often, it's not that the technology is inadequate; you just haven't used it correctly yet. Optimize what you have before switching databases.

Requirement 5: "Add a Scheduled Task, Run It Every Morning at 8 AM"

Background

This requirement seems the simplest — just add a scheduled task. But it's the starting point of every backend developer's nightmare.

I've encountered these "scheduled tasks" over time:

"Sync data once every day at 2 AM" — then the data source went down, the full sync failed, and the entire company had inconsistent data the next day.
"Clean up expired data every hour" — then the cleanup logic had a bug and deleted data that hadn't expired yet.
"Check service status every minute" — then the check endpoint itself had a performance issue, hitting it every minute brought the server down.

Technical Analysis

The biggest pitfall of scheduled tasks isn't "how to write them," but what happens when they fail.

A comprehensive list of scheduled task pitfalls:

1. Task execution time exceeds the scheduling interval → Task pile-up, system crashes
2. Task execution fails → No one knows, data remains problematic
3. Duplicate execution in a distributed environment → The same task runs multiple times
4. Server clocks out of sync → A task scheduled for 8 AM runs at 7 AM on some servers
5. No idempotency → Retries cause duplicate data
6. No timeout control → Task hangs, consuming resources

My Solution

# A reliable scheduled task framework
import asyncio
from datetime import datetime, timedelta
import logging
from functools import wraps

logger = logging.getLogger(__name__)

def scheduled_task(
    name: str,
    max_runtime: int = 300,  # Maximum execution time (seconds)
    max_retries: int = 3,
    retry_delay: int = 60
):
    """Scheduled task decorator: automatically handles failures, timeouts, and notifications"""
    def decorator(func):
        @wraps(func)
        async def wrapper(*args, **kwargs):
            start_time = datetime.now()

            for attempt in range(max_retries + 1):
                try:
                    # Timeout control
                    result = await asyncio.wait_for(
                        func(*args, **kwargs),
                        timeout=max_runtime
                    )
                    # Log success
                    elapsed = (datetime.now() - start_time).total_seconds()
                    logger.info(
                        f'[Scheduled Task] {name} executed successfully '
                        f'(Duration: {elapsed:.1f}s, Retries: {attempt})'
                    )
                    return result

                except asyncio.TimeoutError:
                    logger.error(
                        f'[Scheduled Task] {name} execution timed out '
                        f'(exceeded {max_runtime}s, attempt {attempt+1})'
                    )

                except Exception as e:
                    logger.error(
                        f'[Scheduled Task] {name} execution failed: {e} '
                        f'(attempt {attempt+1})'
                    )

                if attempt < max_retries:
                    await asyncio.sleep(retry_delay)
                else:
                    # Retries exhausted, send alert
                    await send_alert(
                        f'Scheduled task {name} failed {max_retries+1} consecutive times. Please check!'
                    )
                    raise

        return wrapper
    return decorator

# Usage example
@scheduled_task(name='sync-user-data', max_runtime=600, max_retries=3)
async def sync_user_data():
    """Sync user data"""
    # 1. Acquire distributed lock (prevents duplicate execution)
    lock = await acquire_lock('sync-user-data', ttl=600)
    if not lock:
        logger.info('Previous task is still running, skipping')
        return

    try:
        # 2. Idempotency: Use a batch ID to ensure the same data isn't processed twice
        batch_id = f"batch_{datetime.now().strftime('%Y%m%d%H%M%S')}"
        # 3. Paginated processing, record progress per batch
        # 4. Exception handling: A single row failure doesn't affect the whole batch
        pass
    finally:
        await release_lock(lock)

Three Iron Rules

After suffering so many losses, I set three iron rules for scheduled tasks:

Must include alerts: Task failure = automatic notification. Don't wait for users to discover it.
Must be idempotent: Retries must not cause duplicate data.
Must be observable: Logs, duration, success rate — none can be missing.

Final Words: The "Simple" and "Complex" of Backend Development

Looking back at these bizarre requirements, I've noticed a pattern:

The "simple" that stakeholders talk about and the "simple" that developers understand are two completely different concepts.

Stakeholders' understanding of "simple" = Short requirement description, sounds intuitive. Developers' understanding of "simple" = Clear technical solution, no pitfalls, small scope of changes.

The gap between these two is where backend developers most often fall into traps.

Advice for fellow developers:

"Add a field": First ask what this field is for, and what tables, APIs, caches, and scheduled tasks it might involve.
"Export function": First ask about the data volume, then design the solution.
"Switch databases": First ask about the pain points, optimize the existing solution, then consider migration.
"Just get it online first": Security is the foundation; it can't be patched up later.
"Scheduled task": Alerts, idempotency, observability — none can be missing.

Finally, a word for product managers:

When you think a requirement is very simple, please believe — it is definitely not simple. If it truly is simple, it means you haven't fully thought through what it's supposed to do.

What backend requirements have spiked your blood pressure? Feel free to share in the comments. I'd also love to see everyone's "late-night log debugging" moments 😂

#BizarreRequirementsCollection #BackendDevelopment #ProgrammerLife #Database #TechSharing

Comments

Top 1 from juejin.cn, machine-translated. The original thread is authoritative.

LGGGGG

Programmer: Open AI, enter prompt: Add an xxx field to the database, handle all the places where it's used.