AI Backends Are Just a Fleet of Containers

Direction: AI application development backend Writing date: 2026-08-18

I. Origin: Why someone aiming for AI backend needs to learn Docker first

I set my future direction as AI application development backend. In many people's imagination, "AI backend" means calling model APIs, writing prompts, and building RAG. But after really starting to do it, I found that a usable AI application, at the engineering level, is actually a bunch of services collaborating: model inference services, vector databases, conversation caches (Redis), user and business data (MySQL), gateways and proxies…

Each of these services has strict environment and version requirements. Python needs 3.10, CUDA needs a specific version, Node needs 16, Redis needs 7. Installing them on one machine causes version conflicts and environment pollution—this was the first wall I hit.

So I asked myself a question: In those large-scale production environments, how is the infrastructure for AI applications actually set up? Following this question, I arrived at Docker.

II. Lesson One: Container thinking — what Docker really is

When learning Docker, the first thing that made it "click" for me was this analogy:

Docker is like a giant cargo ship and shipping containers.

A giant cargo ship can transport thousands of containers at once. Each container holds completely different goods—some hold cars, some hold clothing, some hold fruit. They don't interfere with each other, get loaded onto the ship, cross the ocean, arrive at any port, and are ready to use as soon as they're unloaded.

Our applications are the same. In the past, delivering software meant "handing over a pile of code + a pile of environment requirement documents." The deployer still had to configure the environment on-site, match versions, and troubleshoot. Docker packages code and runtime environment together into a standardized container. Any machine with Docker installed can pull it and run it, just like a container can be hoisted at any port.

I summarized Docker's essence in one line:

Agent  = LLM + Harness(tool + mcp + rag + skill + ...)
Docker = Application + Runtime Environment

This comparison is a small insight of my own. When working on AI Agents, I understood that "an Agent is not just a model, but the model plus a whole set of tools, context, and skills loaded together." The same mental model applies to Docker—an application is not just code, but code plus the entire runtime environment it depends on. Packaging this "whole" is exactly what containerization does.

III. The pitfalls I encountered: Why "isolation" is needed

Principles are empty; pitfalls are real. What really made me want to do this was a real scenario close to home:

You join a company and take over a Vue2 project written n years ago, requiring Node16 + npm 8. But your computer has Node 22 installed, and the code errors out as soon as you run it.

This isn't a rare case; it's backend daily life. Java has JDK version wars, Python has the 2/3 split, Node has rapid version iteration, not to mention the dependency hell of the C family. Every version difference is an environment conflict; every environment conflict is a round of unpaid overtime.

Containerization technology installs dependencies in isolation: one container per project, each with its own Node version, its own dependencies, not bothering each other. You don't need to uninstall and reinstall Node on your own machine for an old project—just like a giant cargo ship won't turn the entire ship into a cold storage just because the fruit in one container needs refrigeration.

This idea of "isolation" was very important for my later understanding of service governance in AI backends: model services, vector databases, business databases are essentially independent containers, running in their own containers, collaborating through ports and networks.

IV. Core concepts: Image and Container

Once you understand the "shipping container" mental model, Docker's two most important concepts fall into place:

• Image: A packaged, read-only template containing the application and environment. I compare it to git pull—pulling code from a remote repository. Docker pulls an image from an image registry.
• Container: A running instance of an image. I compare it to a DVD—the image is like a master disc, which can be copied many times, each copy running as an independent container; or, the image is like a disc template, and run burns a playable "player".

# Pull image (analogy: git pull)
docker pull nginx

# Run image, becoming a runnable container
docker run --name my-nginx-demo -d nginx

# Stop all containers / Remove all containers / Remove image
docker stop $(docker ps -q)
docker rm   $(docker ps -aq)
docker rmi  <image name>

Another point that confused me at first: ports.

When we visit a website and type www.juejin.cn:3000, the :3000 after the domain name is the port. The default :80 is the default port for HTTP, so visiting www.example.com actually visits www.example.com:80, but the browser hides it for us.

In Docker, containers are isolated, so external access can't get in. Therefore, port mapping is needed: map a port on the host machine (our computer) to a port inside the container. For example:

docker run --name my-nginx-demo -p 80:80 -d nginx
#          ↑ container name     ↑ host 80 port : container 80 port

-p 80:80 means "the host's port 80 corresponds to the container's internal port 80". When a user's browser enters http://localhost:80, the request is forwarded and mapped into the container's port 80.

V. Hands-on practice: Deploying a Node service with nginx reverse proxy

No amount of theory beats running it once. Following my notes, I did the simplest but complete practice: a Node service + nginx reverse proxy.

First, write a bare-bones Node service, listening on port 1314, returning "hello world":

// demo/index.js
const http = require('http');
const server = http.createServer((req, res) => {
  res.end("hello world");
});
server.listen(1314, '0.0.0.0', () => {
  console.log('node server run on 1314');
});

In demo/conf.d/nginx.conf, configure nginx reverse proxy to forward requests from port 80 to the Node service on port 1314:

# demo/conf.d/nginx.conf
server {
    listen 80;
    location / {
        proxy_pass http://host.docker.internal:1314;
        proxy_set_header Host $host;
        proxy_set_header X-Real-IP $remote_addr;
    }
}

Then start the nginx container, mounting the local configuration into it:

docker run `
 --name my-nginx-demo `
 -p 80:80 `
 -v D:\workspace\sw_ai\backend\docker\demo\conf.d:/etc/nginx/conf.d `
 -d nginx

Breaking down this command:

Parameter	Function
--name my-nginx-demo	Names the container
-p 80:80	Maps host port 80 to container port 80
-v host_dir:container_dir	Mounts the local nginx config file into the container; changing config doesn't require rebuilding the image
-d nginx	Runs the nginx image in the background

After running it, the complete request chain is:

User's browser (chrome)
   → localhost:80 (forward proxy out to network)
   → docker -p port mapping → inside container :80
   → nginx receives access on port 80
   → per config file, reverse proxies to :1314 (host.docker.internal points to host)
   → Node service returns "hello world"

VI. Reflective thinking: What reverse proxy really means

This demo looks simple, but it made me understand a key backend concept: reverse proxy.

When a user accesses localhost:80, the browser (forward proxy) helps us get out to the network; and nginx on port 80 "receives requests on behalf of the backend server"—the user has no idea what port or which machine the backend is actually running on. This action is called reverse proxy.

Its significance goes far beyond "forwarding":

1. Hides the backend cluster: The real backend services hide in the internal network; only port 80 is visible externally, much safer.
2. Handles high concurrency: nginx can handle very high concurrency, distributing requests by rules to multiple backend instances behind it—this is the prototype of load balancing.
3. Scalable: Adding or removing a backend machine is completely transparent to the frontend.

Ops knowledge: The server software proxies all requests on port 80 to port 3000 (or 1314).

This pattern of "one external port, a group of internal services" is the standard posture for all modern backends (including AI backends).

VII. Docker's place in my AI backend roadmap

This is the part of the blog I most want to talk about. Learning Docker isn't just about supplementing some ops knowledge; it's that I found the entire infrastructure of AI application backends can almost all be understood using this "shipping container" thinking:

1. Model-as-a-Service: The LLM itself is a container

In a production environment, you almost never directly import a hundred-billion-parameter model in your code. Usually, model inference is encapsulated as an independent model service (e.g., a container running vLLM / Ollama), exposing an HTTP interface externally, which the business backend calls. This is exactly the same model as the Node service + nginx above—the model service is just a 1314 hiding behind the proxy.

2. MCP / Agent toolchain: Every tool can be containerized

My understanding of Agent is LLM + Harness(tool + mcp + rag + skill...). These tools and MCP servers are essentially independent services. Containerizing each of them allows independent upgrades, independent scaling, independent deployment—upgrading a tool today doesn't affect the main service, just like damage to one container's cargo on a ship doesn't affect other containers.

3. RAG infrastructure: Redis + MySQL + Vector DB

A RAG application has cache (Redis), business data (MySQL), and a vector database hanging behind it. I first experienced this when installing mysql with Docker in my notes:

docker exec -it mysql-demo /bin/bash   # Enter container
mysql -uroot -p123456                   # Connect to mysql
create database blog;                    # Create database

These "database services" require absolutely nothing to be installed on my own computer—pull an image, start a container, done. My laptop stays clean, yet can run multiple versions of MySQL, Redis, Postgres simultaneously. This is the value of isolation.

4. The ultimate solution for environment consistency

The environment hell in the AI field is even worse than regular backends: CUDA versions, Python versions, inference frameworks (PyTorch / vLLM / llama.cpp)… Everyone's machine is different, and the same prompt might produce different results in different environments. Docker locks the entire runtime environment (including GPU driver dependencies, CUDA versions) into an image, so every step of development and production is in the same "container", eliminating "it works on my machine" at the root.

VIII. Next steps

After learning Docker basics, my roadmap is:

1. Complete Docker practice: Use docker-compose.yml to orchestrate Node + Redis + MySQL + a model service all at once, building a "production-like" local AI backend.
2. Write a real AI application backend: Based on NodeJS (my main backend framework), implement a service with RAG, containerizing the model service, vector DB, and Redis cache to run.
3. Deep dive into MCP and Agent servitization: Package self-developed tools as MCP servers, deploy with Docker, and experience the backend form of a "pluggable toolchain".
4. Understand GPU containers and production deployment: Study how model services run and scale in GPU-equipped containers, preparing for future production deployment.

IX. Conclusion

Looking back, the first lesson Docker taught me wasn't a specific command, but a way of thinking: break a complex system into a bunch of "shipping containers" that don't interfere with each other but can collaborate.

This mental model happens to be isomorphic to what I understood when building AI Agents—Agent is LLM + Harness, Docker is Application + Environment, both are holistic packaging of "core + supporting load". And the AI application backend, in the final analysis, is about rationally orchestrating these containers and making them work together.

From writing a Node service that returns hello world over a cup of coffee, to containerizing and running nginx reverse proxy, mysql, and redis one by one—the road is still long, but the direction is getting clearer and clearer.

What we load into containers is not just code, but the infrastructure on which every future AI application will depend.