AIOps (Part 1): Building AI-Powered Operations with Prometheus MCP + WorkBuddy

Based on a Linux monitoring scenario, this article uses Docker Compose to deploy Prometheus, Node Exporter, and Prometheus MCP Server, then connects them to WorkBuddy to query monitoring metrics in natural language and automatically generate HTML visual inspection reports.

AIOps intelligent inspection report effect

1. Overall architecture

WorkBuddy + Prometheus MCP architecture

Component	Main responsibility
Node Exporter	Exposes Linux CPU, memory, disk, network, and load metrics
Prometheus	Scrapes and stores metrics, provides queries via PromQL
Prometheus MCP Server	Wraps the Prometheus API as MCP tools callable by AI
WorkBuddy	Understands natural language, calls tools, analyzes data, and generates HTML

MCP (Model Context Protocol): Provides a standardized way for AI applications to call external tools. In this setup, data collection is done by Node Exporter and Prometheus; WorkBuddy queries the collected data through the MCP Server and generates reports.

2. Docker Compose deployment: 3 containers

Prerequisites: a Linux cloud server with Docker Engine / Docker Compose V2 installed, and a local computer with WorkBuddy installed. The examples use public IP 203.0.113.10 (a documentation-reserved address; replace it with the server's actual public IP in deployment).

2.1 Create Prometheus scrape config

On Linux server 203.0.113.10 run:

mkdir -p ~/aiops-prometheus && cd ~/aiops-prometheus
vi prometheus.yml

Save the following to prometheus.yml:

global:
  scrape_interval: 15s

scrape_configs:
  - job_name: prometheus
    static_configs:
      - targets: ['localhost:9090']

  - job_name: node
    static_configs:
      - targets: ['node-exporter:9100']
        labels:
          instance: '203.0.113.10'

Here node-exporter:9100 is the address accessible inside Docker, and instance is used as the monitoring instance identifier.

2.2 Create Compose file

vi compose.yaml

services:
  prometheus:
    image: prom/prometheus:latest
    container_name: prometheus
    restart: unless-stopped
    ports:
      - '19090:9090'
    volumes:
      - ./prometheus.yml:/etc/prometheus/prometheus.yml:ro
      - prometheus-data:/prometheus

  node-exporter:
    image: quay.io/prometheus/node-exporter:latest
    container_name: node-exporter
    restart: unless-stopped
    volumes:
      - /proc:/host/proc:ro
      - /sys:/host/sys:ro
      - /:/rootfs:ro,rslave
    command:
      - --path.procfs=/host/proc
      - --path.sysfs=/host/sys
      - --path.rootfs=/rootfs
    expose:
      - '9100'

  prometheus-mcp-server:
    image: swr.cn-north-4.myhuaweicloud.com/ddn-k8s/ghcr.io/tjhop/prometheus-mcp-server:latest
    container_name: prometheus-mcp-server
    restart: unless-stopped
    depends_on:
      - prometheus
    ports:
      - '8080:8080'
    command:
      - --prometheus.url=http://prometheus:9090
      - --mcp.transport=http
      - --web.listen-address=:8080

volumes:
  prometheus-data:

This configuration starts 3 containers total: Prometheus is exposed externally on port 19090, the MCP Server is exposed on 8080, and Node Exporter is scraped by Prometheus through the Docker internal network. Prometheus data is stored in a persistent volume.

On the cloud server, allow TCP 8080 and 19090: for Alibaba Cloud Lightweight App Server configure them in the firewall; for ECS use a security group. It is recommended to allow only trusted source IPs, and production environments should also configure HTTPS and authentication.

2.3 Start and verify

docker compose up -d
docker compose ps
curl -sG http://127.0.0.1:19090/api/v1/query --data-urlencode 'query=up'

Confirm all three containers are Up, and that in the query result both job="prometheus" and job="node" have value 1, meaning the metric collection path works.

3. Connecting WorkBuddy to Prometheus MCP

Open Experts · Skills · Connectors → Connectors → Custom Connector → Add MCP.

Enter the connector page

Add a custom MCP

Fill in the following JSON, replacing the address with the actual cloud server public IP:

{
  "mcpServers": {
    "prometheus-mcp": {
      "type": "http",
      "url": "http://203.0.113.10:8080/mcp"
    }
  }
}

Fill in MCP connection config

After saving and confirming trust, expand the MCP tool list:

Prometheus MCP tools loaded successfully

This test successfully loaded 28 MCP tools, including query, range_query, list_targets, etc. The exact number depends on the version and configuration.

4. Create an AIOps visual inspection expert

Go to Experts · Skills · Connectors → Experts → My Experts → Create Expert, and associate prometheus-mcp.

My Experts

Create expert

Here is the complete expert prompt that can be copied directly:

You are an SRE/AIOps intelligent operations expert, familiar with Linux, Docker, Kubernetes, Prometheus, PromQL, and MCP. You have the prometheus-mcp tool attached.

[Goal]
Based on real Prometheus metrics, complete resource inspection, trend analysis, and anomaly diagnosis. Prioritize delivering Grafana-style HTML visual reports, and keep the text summary concise.

[Query requirements]
1. First query targets, job, instance, and metric to confirm the collection targets and available metrics.
2. By default analyze the last hour, with range_query step 60 seconds; when the user specifies otherwise, follow the user's specified range.
3. Each MCP query/range_query submits only one valid PromQL statement.
4. CPU, memory, disk, network, and system load must be calculated from real metrics; do not simulate, fabricate, or fill in missing data.
5. When metrics are missing, targets are DOWN, or the time window is incomplete, explain the reason and the valid data range truthfully.

[Analysis requirements]
1. Show current value, average, peak, and trend; distinguish facts, concerns, and hypotheses to be verified.
2. Do not determine the root cause of a fault from a single metric, and do not simply treat a short-term peak as sampling error.
3. Zero active alerts does not mean the alerting system is sound; thresholds must be labeled as actual rules or suggested values.
4. Read-only by default: do not arbitrarily restart, delete, change configuration, or auto-fix.

[Visual delivery]
If the environment supports file generation, actually create Prometheus_AIOps_巡检报告.html and provide a preview entry. It is recommended to use standalone HTML + CSS + SVG/JavaScript, including:
- Overview KPIs: host identifier, exporter health, CPU, memory, disk, network, load;
- Real trend charts: CPU, memory, disk, network RX/TX, Load 1/5/15, with time axis and units labeled;
- Resource details: total, used, available, partitions, and network interface names;
- AI diagnosis: anomaly evidence, risk description, recommended troubleshooting commands;
- Inspection conclusion: within 100-200 characters, highlighting risks and next actions.

[Calculation and display rules]
CPU uses the rate of the idle counter; memory uses MemAvailable/MemTotal; disk uses avail/size, excluding virtual file systems. Accurately distinguish GB/GiB, Bytes/s and bit/s. Display times in Beijing time UTC+8. All chart points must correspond to real query results; do not use simulated samples, and clearly mark missing data.

Please indicate that the HTML is a static inspection snapshot, not an auto-refreshing real-time dashboard. Finally, output only a short summary and the actual generated file entry; if the file cannot be created, say so honestly and do not pretend it was generated successfully.

Select the created inspection expert

5. Start an inspection and view the HTML dashboard

Send the following instruction to the created expert:

Please use prometheus-mcp to inspect instance="203.0.113.10" for CPU, memory, disk, network, and system load over the last hour. First check the collection status, use real range_query time series data, generate a dark Grafana-style HTML inspection dashboard, including KPI cards, trend line charts, resource details, AI diagnosis, and inspection conclusion. Do not just output Markdown; actually generate the HTML file and provide a preview. Use Beijing time uniformly, clearly mark missing data, and do not fabricate.

Submit the inspection task in WorkBuddy

Monitoring overview: Directly displays KPIs such as CPU, memory, disk, load.

Inspection report KPI overview

Resource trends: Shows CPU and memory changes through real time series.

CPU usage real time series curve

Memory usage real time series curve

AI analysis: Summarizes metric anomalies, diagnostic evidence, and suggested actions.

AI intelligent diagnosis and inspection conclusion

In the sample report, CPU is about 2.4%, memory about 45%, root partition disk about 66.5%. The HTML file is a one-time inspection snapshot, not an auto-refreshing real-time dashboard. The specific values in the sample report only represent this test environment.

References: Original practice article · Prometheus · Node Exporter · Prometheus MCP · Docker Compose