One AI Gateway, Six Layers: The Blueprint for Enterprise AI Testing as Infrastructure

Huge! Another open-source enterprise-grade one-stop AI intelligent testing platform! (with platform architecture and tech stack)

Before National Day, I conducted a survey in AI Evolution Society on the current state of AI usage at students' companies.

A few numbers first.

Nearly 95% of students' companies have started AI transformation, and half mentioned that their company has 'large-scale multi-department use'. But 80% of AI projects actually implemented in enterprises are concentrated in the two lowest-threshold scenarios: knowledge base management and development efficiency. The penetration rate of deep business projects such as RPA, Agent, and intelligent quality inspection is less than 40%. On one hand, almost everyone has started; on the other hand, everyone is collectively stuck in the shallow water.

What is stuck in between? Actually, the answer can be found in the enterprise leaders' requirements for AI efficiency in the survey results: quantified efficiency indicators, business integration, and replacing repetitive human labor.

Specifically for product and R&D roles, 'full-process AI testing' is the most frequently mentioned item. For example, hard quantified indicators such as '80% of test cases are run by AI itself' and 'weekly iteration changed to daily iteration' have become the norm in enterprises.

To sum up in one sentence: AI is the general trend; the top wants results, the bottom lacks capability.

That is the real situation for testing teams in 2026, and it is also the problem that the AI Intelligent Testing Platform Hands-on Special Program is meant to solve.

1024 Programmer's Day is coming soon, and exclusive holiday benefits have been prepared for everyone. See the end of the article~

1. The positioning of this project in AI Evolution Society

AI Evolution Society has so far accumulated a lot of knowledge around AI testing capability building.

For example, in the AI Testing Practical series, we dismantled and verified single-point capabilities in API and UI automation one by one: document intelligent parsing, test data construction, batch script generation, execution scheduling, failure automatic diagnosis, and report generation. Each link was made into an independently running Agent Skill (currently, for the full AI testing process, we have accumulated more than 30 special self-developed Skills).

In the LLM Large Model and Agent Evaluation System column, we have updated more than a dozen tutorials. This column built a complete AI evaluation knowledge system, linking the whole AI evaluation process: from evaluation methodology, evaluation set construction, metric design, and scoring schemes, to selection and practice of mainstream evaluation tools such as Promptfoo, DeepEval, RAGAS, and Langfuse, plus real-project evaluation practice for RAG and Agent. Ultimately it completes the engineering of evaluation capability, connects CI gates and platform landing, and realizes integrated mastery of LLM and AI Agent evaluation capabilities from 0 to 1.

But these contents share a common limitation: most of them are used to solve a specific scenario in daily work. Different scenarios require different tools, methods, and technical means, and there is no unified carrier.

The core goal of our AI Intelligent Testing Platform project is to gather and integrate these scattered capabilities into a unified, reusable enterprise-level solution. It is also the most important comprehensive practical project of AI Evolution Society in 2026. The goal is to lead everyone to build a one-stop AI intelligent testing platform from scratch, covering all scenarios of automation (unit, API, Web, App), performance, security, evaluation, RAG, Agent, and Harness, connecting and realizing deep integration of various testing capabilities with business scenarios.

It is worth mentioning that this project will be serialized and updated as a sub-column of AI Evolution Society. In terms of content format, it adopts a dual form of 'graphic tutorials plus video explanations'.

• Graphic tutorials include comprehensive development details, design ideas, architecture decisions, and complete code (readable even for beginners with zero foundation);
• Video tutorials focus on operation demonstrations and complete hands-on processes, with core knowledge explained by real people. You follow along while watching. The two forms complement each other with the same pace.

2. Why it must be a platform, not a pile of tools

First, look at a scenario many teams are experiencing.

Several backbone members in a team use AI well on their own. One uses large models to generate API test cases, another writes their own script to generate prompts, and another sets up a failure log analysis pipeline. Everyone is improving efficiency, but the capabilities all live on personal computers. Once the backbone member leaves, those capabilities leave with the person, and the team's level regresses overnight. This is the first problem: capabilities are not precipitated.

The second problem is inconsistent quality standards.

For the same requirement, the test cases generated by two people using AI, when put together, differ greatly in granularity, coverage, and sensitivity to abnormal scenarios. In the survey, 60% of enterprises list the talent gap as the first obstacle to advancing AI.

It is not hard for companies to recruit people who can use AI tools; what is hard to find are people who can embed AI into business processes and judge the quality of AI output. Under the decentralized tool model, this judgment cannot be standardized; good or bad depends entirely on personal feel.

The third problem is that management cannot see.

How many test cases AI generated, what percentage of execution is automated, and how many failures can self-heal; these quantifiable indicators that leaders ask for cannot be collected under the decentralized model. Assets are scattered on personal computers, so reports cannot be compiled.

The fourth problem is the most practical: enterprise environments cannot be entered.

In the survey, many students come from government/enterprise and traditional industries: intranet without external network, complex API authentication, and missing documentation for legacy systems. Decentralized tools that run on the public internet and rely on direct calls to cloud large models die on the first step in these environments. A platform that can be privately deployed is the only channel for such enterprises to use AI.

So the essence of platformization is to upgrade 'a few people's AI capabilities' into 'the entire team's infrastructure'. Capabilities precipitate in the platform, quality standards are solidified in the process, metrics grow from platform data, and deployment forms adapt to enterprise environments.

There is another change unique to the AI era worth mentioning. Traditional testing platforms such as ZenTao and TestLink are positioned as management tools; they manage test cases written by people and defects raised by people.

For an AI-era intelligent testing platform, AI itself is the worker: generating test cases, writing scripts, executing runs, and doing diagnosis, while humans retreat to review and decision-making. The work process, review actions, and accumulated data all happen in the platform, and the data in turn makes AI more accurate over time. The platform changes from a 'management tool' into 'the productivity entity itself'. This is the most essential difference between this round of platform construction and traditional testing platforms.

3. Platform architecture design: layered skeleton, plugin scenarios

How is the architecture designed? Platform architecture design is not opening drawing software and directly dragging boxes. The real effort is in the three things before drawing.

The common way to draw architecture diagrams in the industry:

First, sort out requirements: functional requirements, non-functional requirements (performance, availability, security, whether private deployment is supported), and realistic constraints (team tech stack, budget, delivery environment). List them all; every architecture decision afterward must be checked against this list.

Second, choose an architecture style: monolithic layering, microservices, event-driven, plugin-based (microkernel). These are the common types in the industry.

For small and medium scale, choose monolithic layering for simplicity; if functional modules need frequent addition/removal, choose plugin-based for flexibility. No style wins in all cases; it only depends on fit.

Third, define boundaries: decide how many layers, what each layer is responsible for, which direction dependencies point, and what interface contracts look like.

Only after these three steps are clear does drawing begin. No matter how the architecture style changes, several iron laws remain constant: high cohesion and low coupling, single responsibility, and one-way dependencies.

Back to our AI intelligent testing platform. Compared with ordinary management platforms, it has three key characteristics:

• Test scenarios differ greatly. API, UI, performance, security, LLM evaluation, and Agent testing each have their own ways under different types. Forcing them into the same logic will inevitably drag each other down;
• The AI technology stack iterates extremely fast. Today's leading model may be replaced in a few months, and the architecture cannot be rewritten along with the model;
• When deployed in an enterprise, private deployment must be supported; the platform must be able to move into the intranet as a whole.

Around these three characteristics, applying the common approach, the design principles of this project roughly follow four rules:

• The overall structure adopts the layering principle. Each layer answers only one question, and dependencies can only point downward; any layer can be refactored or replaced independently;
• Different scenarios are made into plugins and mounted into the scenario layer, open to extension and closed to modification. A new scenario means adding a plugin, not modifying the main structure;
• The fastest-changing AI capability becomes an independent layer and enters/exits uniformly through the AI gateway. No matter how models change or how cloud/local switching happens, business code is not affected;
• Existing enterprise environments are all placed outside the trust boundary. Tested systems, large models, and enterprise toolchains interact with the platform only through several controlled channels: pipeline triggers, API calls, intranet paths, and execution traffic.

When these four principles are mapped to the diagram, they correspond to the six-layer skeleton, the scenario extension slot, the AI gateway, and the dashed external-environment band on the far right.

Cut according to these principles, the initial form of the platform architecture is the diagram below:

Let us go through the diagram from top to bottom.

L1 User Access Layer: Web console, AI dialogue assistant, CI/CD integration, CLI and OpenAPI. People, scripts, and external systems; all entry points are collected in this layer.

The next layer down is L2 Scenario Service Layer: the six major testing scenarios. On the far right there is also a dashed 'scenario extension slot'. Scenarios that do not exist today can be mounted tomorrow without touching the main structure.

L3 AI Capability Layer: Agent orchestration, prompt and rule library, RAG retrieval, and AI gateway. All intelligence of the platform comes from this layer. It is worth mentioning that today almost all major Internet companies have their own AI gateway services for unified management and scheduling of internal AI capabilities. This has become standard infrastructure in the AI era.

The next three layers are invisible to users.

L4 Scheduling Execution Layer: task scheduling, failure diagnosis and self-healing, and executor cluster. Test tasks actually run on this layer.

L5 Data Layer: business database, cache queue, vector database, and object storage. Four types of data and four types of storage are in place.

L6 Infrastructure Layer: Docker container pool, browser grid, Agent runtime, and monitoring/logging. The five layers above all rest on it.

4. How to divide architecture layers: each layer solves one type of problem

How exactly should architecture layering be divided? Why is the architecture diagram above 6 layers, not 5 or 7?

Conclusion first: more layers is not better.

Each additional layer adds one interface, one call chain, and one troubleshooting path. A request has to pass one more gate; when a problem occurs, logs have to be flipped through several more screens.

Too few layers also does not work. Two things with different responsibilities squeezed into the same layer: changing one affects the other, and touching one part moves the whole body.

The number of layers is never the goal of layering. It is only the result that naturally falls out after responsibility boundaries are clearly cut. If the boundaries are cut correctly, it is however many layers it should be.

What criteria are used for cutting? The most fundamental one: each layer only answers one type of question, and answers it completely.

Applied to this project, there are exactly six types of questions: where to enter, what work to do, who thinks, who does the work, where it is stored, and where it runs.

The six types of questions are each independent and do not overlap. One more layer is too fragmented; one fewer layer squeezes two questions together. Therefore it is 6 layers, not 5 or 7.

The half sentence 'answers it completely' is equally important. For example, permission verification: taking it out as a separate layer sounds simple, but it is part of the 'where to enter' question. Splitting it out instead cuts the entry point apart.

Another equally useful rule: things with different change rhythms are not placed in the same layer.

Layering divides not only responsibilities, but also the speed of change. AI capability is the fastest-changing part of the whole platform; models are replaced every month or two, and prompt strategies are adjusted at any time. User entry points and scenario services are relatively stable. If they are locked in the same layer, every time the fast-moving part moves, the slow-moving part is affected. Many layer boundaries actually fall on the dividing line between fast and slow change.

After cutting with these two criteria, the six layers above are the result. Each layer focuses on solving one type of problem:

1. L1 User Access Layer answers 'where to enter'.

Web console, AI dialogue assistant, CI/CD integration, CLI and OpenAPI. People, scripts, and external systems; all entry points are collected in this layer.

Conversational interaction is the highlight of this layer. The AI dialogue assistant is built on MCP and Skill, using natural language to command the platform to work. It was the most requested requirement at project initiation. Enterprise pipelines also connect from this layer; a Webhook on Jenkins or GitLab can trigger a regression round.

1. L2 Scenario Service Layer answers 'what work to do'.

Each of the six testing scenarios is a service module, sharing the same set of test case assets, task models, and report models. This layer has the lowest presence but the highest extensibility requirements.

1. L3 AI Capability Layer answers 'who thinks'.

Agent orchestration, prompt and rule library, RAG retrieval, and AI gateway; all intelligence of the platform comes from this layer. The significance of making it an independent layer is that changing models, prompt strategies, or Agent frameworks should not affect the business layer.

1. L4 Scheduling Execution Layer answers 'who does the work'.

Task scheduling queues and distributes work, the executor cluster actually runs tests, and failure diagnosis and self-healing handle closing out. Unattended scheduled regressions and nightly inspections rely on this layer.

1. L5 Data Layer answers 'where it is stored'.

Test case assets go into the business database, queues go into cache, knowledge goes into the vector database, and screenshots and reports go into object storage. Four types of data, four types of storage, each in place.

1. L6 Infrastructure Layer answers 'where it runs'.

Docker container pool, browser grid, Agent runtime, monitoring and logging; the running foundation of the platform itself.

There is an iron law between layers: dependencies can only go downward, not upward. The user access layer can call the scenario layer, the scenario layer can call the AI capability layer, but not the reverse. With this rule held, every layer can be independently refactored and replaced without dragging others down.

5. Tech stack combination

Technology selection has basically been finalized at this point, collected into a master list. After the project starts, if you forget, come back here any time:

Category	Selection	Purpose
Backend language	Python	Main language for the whole platform
Backend framework	FastAPI	Async native, holds high concurrency model calls
AI application framework	LangChain + LangGraph	Agent orchestration and RAG
Unified model access	LiteLLM	Embedded in the self-developed AI gateway, unified access to hundreds of models
Frontend	TypeScript + Vue 3 + Vite + Ant Design Vue	With Pinia, ECharts, Monaco Editor
Business database	PostgreSQL	Primary storage, pgvector also for vector retrieval
Cache	Redis	Cache, rate limiting counters, distributed locks, temporary state
Async task queue	RQ (Redis Queue)	MVP main line; interface designed according to Celery concepts
API testing	HTTPX + pytest + jsonschema	API execution engine base
UI testing	Playwright	UI execution engine base, reused for Agent testing
Performance testing	Locust	Python-native load testing
Security testing	OWASP ZAP + self-maintained rule library	General security scanning + targeted checks
AI testing	DeepEval + Midscene.js	Use DeepEval to test AI, use Midscene to let AI test
Container orchestration	Docker Compose (K8s reserved)	Single-machine main line, sufficient for private delivery
Reverse proxy	Nginx	Static hosting, API reverse proxy, HTTPS
Monitoring and logging	Prometheus + Grafana + Loki	Check monitoring, view logs
Others	Add as needed later

6. AI gateway: the master switch of platform intelligence

The AI gateway in the architecture diagram deserves to be pulled out and discussed first. This section explains why it deserves this treatment.

First, look at the world without a gateway.

Each scenario service directly connects to the large model on its own: the API testing module writes a piece of call code, the UI testing module writes another, and the LLM evaluation module writes yet another. Soon you will find problems: API keys are scattered everywhere and permissions cannot be centrally controlled; if you want to count how many tokens AI called this month and how much money was spent, the data cannot be gathered; one day you want to change model vendors, and the whole platform code has to change. The most fatal one is when intranet customers arrive and require all inference to go through local models, and you find there is no place where the 'switch to intranet' action can be executed.

The AI gateway is the component that closes all these problems at once. It exposes only one set of OpenAI-compatible protocols to the outside. This protocol is now the de facto industry USB-C: cloud large models, local vLLM, and Ollama all speak this protocol. All components in the platform recognize only the gateway, not a specific model:

• Multi-model intelligent routing: generating test cases routes to cheap and fast models, failure diagnosis routes to strong-reasoning models, allocated by scenario, with the balance of cost and quality in your own hands;
• Privatization switch: in intranet deployment, the gateway switches entirely to local inference, and the upper layers of the platform do not need to change a single line of code. This is the meaning of the dashed 'intranet path' in the architecture diagram;
• Governance and metering: retry, degradation, and rate limiting are done uniformly at the gateway, and usage data for every call is written to storage and fed directly to management dashboards. The 'AI efficiency can be measured' mentioned in the previous article; the data source is here.

In this project, we will build a lightweight AI gateway ourselves rather than directly applying a ready-made solution such as LiteLLM. The reason is straightforward: the routing, degradation, and metering logic inside the gateway is one of the most worth-learning parts of this platform. Wrapped in someone else's library, it cannot be learned. The interface is implemented according to protocol standards, so if production needs to switch to a ready-made solution, it can be replaced at any time.

7. What this platform means to enterprises

From the enterprise perspective, investing resources to build a testing platform actually buys two things.

One is efficiency. AI takes over the large human-resource tasks of test case writing, script development, regression execution, and report output, and people focus on business understanding and risk judgment.

The other is certainty. The process of efficiency improvement is traceable, results are measurable, and capabilities can be retained.

The former determines whether this is worth doing; the latter determines whether it can continue to be done. The significance of this platform to enterprises lies more in the latter.

First, the most direct point: efficiency improvement becomes measurable.

Every efficiency-improvement action in an enterprise ultimately has to answer the same question: how much was saved, how much faster, and whether quality dropped. The decentralized AI model cannot answer this question. Even if it can, it is purely subjective, with data scattered on personal computers, making measurement difficult and reporting impossible.

The platform can leave traces of the entire test activity. AI test case generation ratio, automated execution ratio, failure self-healing rate, regression duration changes: open the panel and there are ready-made numbers. Management has a basis for setting goals, teams have data for reporting progress, year-end ROI has accounts to check. A goal like '80% of test cases run by AI itself' can only be verified on a platform.

Going one level deeper, full-process AI transformation has a unified carrier.

Testing work is scattered across requirements analysis, test case design, script development, execution regression, and report output. The platform collects the entire chain into one entry: natural language goes in, test assets come out; before test submission, automation is ready; smoke tests run automatically; reports are generated automatically; there is no manual transfer between tools.

At the same time, the testing platform stands in the middle of the enterprise toolchain, connecting upward to requirements and defect management, and downward to tested systems and CI pipelines. AI output flows directly back into existing enterprise tools, and teams do not need to overturn existing processes to use AI.

Third, experience changes from personal assets to organizational assets.

Prompts, generation rules, review standards, and diagnosis strategies are all version-managed in the platform. Senior engineers' judgment standards are written into rules; new people using the platform automatically inherit these standards. Backbone personnel turnover no longer takes away team capabilities, and quality standards no longer vary by person. This is something the decentralized model can never achieve.

Finally, private deployment.

The first three layers of value were originally unreachable for teams in government/enterprise, finance, and other intranet environments; data leaving the intranet is a red line. The platform supports localized private deployment, and large models can be replaced with local models or an intranet inference gateway. Realistic constraints such as no external network, complex API authentication, and missing legacy documentation all have adaptation plans. AI capabilities can get in, and enterprise data cannot get out.

Taken together, what this platform gives enterprises is one certain thing: bringing testing teams from 'everyone improving efficiency individually' into 'the whole team improving efficiency at scale', and making every part of the improvement visible, calculable, and retainable.

8. What it can bring to individuals

If the enterprise perspective looks at organizational capability, the individual perspective should see opportunity.

First, look at the most realistic gap.

When applying for test development positions now, candidates whose resumes say 'skilled in using AI tools to assist testing' are everywhere, and the weight of that phrase is rapidly depreciating. In contrast, 'led the development of an enterprise-level AI intelligent testing platform, covering API, UI, performance, security, RAG, and Agent scenarios' is a completely different magnitude. The former means you have used it; the latter means you have built it. After following this column, you belong to the latter.

Then look at interviews. Every module in platform development directly corresponds to high-frequency test development interview points.

• Doing execution scheduling, you need to answer how task orchestration and resource management are done;
• Doing failure self-healing, you need to answer how to manage flaky cases;
• Doing platform architecture, you need to answer module layering and extensibility design.
• ...

These happen to be the points of differentiation repeatedly emphasized in the interview question series. People who have built a platform answer these questions by describing their real decisions and pitfalls; people who have not can only recite concepts. An interviewer can hear the difference immediately.

Then there is the growth path.

Many functional testing people are stuck between 'clicking' and 'test development'. What they lack is never a language course, but a complete engineering carrier for a project.

This platform is that carrier. You will be forced to face all the problems in real engineering: how to layer architecture, how to orchestrate Agents, how to provide fallbacks when large model output is unstable, and how to adapt to intranet environments. The process of solving these problems is the process of growing from functional testing to AI full-stack test development.

Finally, it is a portfolio piece you can show.

At an interview, open your own platform and demonstrate on the spot the full process from a natural-language requirement to automated execution and report output. This is more convincing than any self-introduction.

9. Final words

Finally, I also want to make it clear: the focus of this intelligent testing platform project is not to produce a finished platform product.

The real value of this project is to provide everyone with a real training ground. It is more like a practical carrier of knowledge: putting engineering development capability, cutting-edge AI technology, and our daily real business testing demands into the same project for a closed-loop implementation.

We are not simply relying on AI to write code and 'pile up a system'. Instead, in the whole process of building the platform from scratch, we learn and practice how to combine engineering capability and AI capability, and face those trivial, thorny, and long-troubling real problems in work.

Code and platform are only carriers. What is really valuable is this integrated problem-solving mindset. After completing the whole project, what you gain is not a repository of source code, but a transferable quality engineering mindset integrated with AI.

From then on, whether you do automated regression, RAG evaluation, Agent task evaluation, or traditional testing scenarios such as performance and security, you can carry this approach and independently design, build, and implement your own solutions.

Later, we will continue to break down every module of the platform, step by step, and lead everyone to iterate and work together to fully land it. See you in practice.