LightLog Replaces the ELK Stack with PostgreSQL and Adds an AI Troubleshooting Agent

Foreword

I've been thinking about how to build a lightweight system that is non-intrusive to business logic yet allows developers/ops personnel to easily troubleshoot problems. There are many solutions on the market, but none seem to fit my needs.

Originally, some of my services used ELK for log collection. A single ELK setup requires at least 3 nodes + 4-8GB of memory, and with Kibana, the resource consumption is heavier than the business processes combined. The operational cost is not low either: ES version upgrades, index templates, ILM policies, disk watermarking, cluster state synchronization—there's a lot to handle.

I tried Loki for a while. It's genuinely lightweight, but full-text search is basically crippled. It only indexes labels, not content. If you want to grep a keyword inside a message, it actually scans all chunks matching the label. When log volume increases, it gets so slow you want to smash your keyboard.

SaaS solutions like Datadog are hassle-free, but expensive. For small to medium projects, it costs tens to hundreds of thousands of yuan a year, which is unaffordable.

The Nasty Status Quo

What's more of a headache is the integration. My own product lines and project lines have distinct characteristics:

• New services on Java 17 + SpringBoot 3.2
• Old services on Java 8 + SpringBoot 2.0, which cannot be migrated

To get logs from these different sources into the same storage, you need to configure three sets of agents: Logstash + Filebeat + OTel Collector, each with its own configuration pitfalls.

So I decided to write my own. Design goals:

• No dependency on ES; use PG for storage (and incidentally verify PG's capability to handle log scenarios)
• A single Java integration method that works across Java 8/17 + SpringBoot 2.x/3.x
• Resource usage ~1GB single process
• Support OTLP standard endpoints, zero-change integration for non-Java applications
• Built-in AI assistant, natural language log querying—in the AI era, this is a must-have—automatically analyzes your service logs to discover potential issues

Take a Look First

Dashboard overview page: KPI numbers + 24h trends + service health table + recent ERROR scrolling.

Logger logs directly input by users in the service, as well as service startup logs and exception stack traces.

Deep Dive into the Tech

Let's expand on a few core technical points.

Storage Layer: Why I Replaced ES with PG

Multi-dimensional filtering by time/service/POD/level/keyword, 32,764 log entries returned in 34ms.

The Cost of ES

ES is powerful, but the cost is high:

• Memory: Recommended 16GB+ per node, half the heap memory for JVM, half for OS page cache (for Lucene)
• Disk: Inverted index itself + source field (original JSON) + doc values (columnar storage), actual usage is 3-5 times the original log size
• Operations: Index templates, ILM policies, shard balancing, cluster state synchronization, version upgrades—each is a pitfall

For small to medium projects, setting up an ES cluster has a terrible ROI.

How PG Handles Log Storage

PG wasn't designed for logs, but several features are sufficient for this scenario:

1. Daily Partitioning

CREATE TABLE lightlog (
    ts          TIMESTAMPTZ NOT NULL,
    service     TEXT NOT NULL,
    instance    TEXT,
    level       TEXT NOT NULL,
    logger      TEXT,
    message     TEXT NOT NULL,
    stack       TEXT,
    mdc         JSONB,
    log_fp      TEXT,
    parsed_meta JSONB
) PARTITION BY RANGE (ts);

CREATE TABLE lightlog_20260805 PARTITION OF lightlog
    FOR VALUES FROM ('2026-08-05') TO ('2026-08-06');

Benefits:

• Cleaning old logs = DROP PARTITION, millisecond-level, unlike DELETE which causes bloat and requires VACUUM
• Queries with time ranges (almost all log queries have them) benefit from PG's partition pruning, which directly skips irrelevant partitions

2. BRIN Index

Logs are written sequentially (monotonically increasing timestamps). Using a B-tree index on such data is wasteful—the index itself takes up space, roughly 1:1 with the data volume.

BRIN (Block Range Index) only stores the min/max of each data block, making the index size 1/100th of a B-tree:

CREATE INDEX idx_lightlog_ts ON lightlog USING BRIN (ts);

The trade-off is that queries need to do "block scanning," reading some blocks outside the range. But log queries almost always include a time range, partition pruning already skips most partitions, and BRIN scans within the remaining partitions with sufficient precision.

3. JSONB + GIN Index

MDC and parsed_meta are semi-structured fields, stored as JSONB, with a GIN index added:

CREATE INDEX idx_lightlog_mdc ON lightlog USING GIN (mdc jsonb_path_ops);

-- Query
SELECT * FROM lightlog WHERE mdc @> '{"traceId": "abc123"}';

Actual test: 30 million log entries, queried by time + service + keyword, results in 34ms. Good enough.

Pluggable storage: SQLite / PG / H2, switchable via environment variables. SQLite for local development (zero dependencies), PG for production, H2 for testing. Spring abstraction + factory pattern, SchemaInitRunner selects the implementation at startup based on LIGHTLOG_STORAGE.

Async Collection + WAL to Prevent Loss: Zero Wait for Business Threads

Async collection is standard for log systems, but implementation details determine reliability.

Data Flow

log.info("xxx")
    ↓ (synchronous, nanosecond-level)
Appender.append()
    ↓ (ConcurrentLinkedQueue.offer, nanosecond-level)
Memory Queue (default 10000)
    ↓ (background thread batch fetch, 1000 entries per batch or 100ms timeout)
WAL File (local disk)
    ↓ (HTTP POST)
Center (HTTP API → insert into PG)
    ↓ (200 OK)
Delete WAL File

The business thread never waits for the network. If the center goes down or the network jitters, the business is unaffected.

WAL Implementation

The core of WAL is "write to disk first, then push." Simplified version (actual business logic is not exactly this):

class WalStore {
    private final Path baseDir;
    private final AtomicInteger counter = new AtomicInteger();
    
    Path append(List<LogEvent> batch) throws IOException {
        String name = "wal-" + System.currentTimeMillis() + "-" + counter.getAndIncrement();
        Path file = baseDir.resolve(name);
        try (OutputStream os = Files.newOutputStream(file)) {
            for (LogEvent e : batch) {
                os.write(serialize(e));
            }
        }
        return file;
    }
    
    void ack(Path file) throws IOException {
        Files.delete(file);
    }
}

// Background push thread
while (!stopped) {
    List<LogEvent> batch = drain(queue, 1000, Duration.ofMillis(100));
    if (batch.isEmpty()) continue;
    
    Path walFile = wal.append(batch);   // Write to disk first
    try {
        centerClient.post(batch);       // Push
        wal.ack(walFile);               // Delete on success
    } catch (Exception e) {
        log.warn("push failed, will retry next round: {}", e.getMessage());
        // Don't delete, retry this file in the next loop
    }
}

The code is under 200 lines. I later wondered if this was over-engineering—for lightweight scenarios, direct synchronous push might be fine. But one detail convinced me: once a user discovers lost logs, trust is broken. A log system can be slow, but it cannot lose data. WAL is not over-engineering; it's the minimum promise of a log system.

Replay on Startup After a business process restarts, the WAL directory may still have unpushed files. The Starter scans the directory on startup, sorts by filename timestamp, pushes them one by one to the center, and only allows new logs into the queue after pushing is complete.

Detail: Startup replay cannot block Spring startup for too long (otherwise the business won't start). So replay is asynchronous, running in a separate thread after the ApplicationReadyEvent.

AI Assistant: Why LangGraph + litellm

Asking "What's the error log situation for ths-api today?" pulled 246 ERROR entries, identified that 95% shared the same fingerprint from ZwUploadServiceImpl, and pinpointed "continuous failure on the same logical path."

Why LangGraph Instead of a Raw Prompt

LangGraph is, after all, LangChain's next-generation agent framework, centered on an explicit state machine + ReAct loop.

Problems with raw prompts:

• LLM can get stuck in infinite loops, burning through tokens
• Tool call process is opaque, making debugging difficult
• Multi-turn conversation state management is chaotic

LangGraph solutions:

• Each node is an explicit function (state → state)
• Edges are state transition rules
• recursion_limit=10 hard-limits the number of loops, preventing token explosion
• Tool call results flow as state, each step can be printed out for inspection

Rough agent structure:

from langgraph.graph import StateGraph, END

builder = StateGraph(AgentState)
builder.add_node("plan", plan_node)        # Parse user question, decide which tools to call
builder.add_node("execute", execute_node)  # Execute tool calls
builder.add_node("analyze", analyze_node)  # Synthesize tool results, generate response
builder.set_entry_point("plan")
builder.add_edge("plan", "execute")
builder.add_conditional_edges("execute", should_continue, {
    True: "execute",   # Continue calling tools
    False: "analyze",
})
builder.add_edge("analyze", END)

graph = builder.compile()
graph.recursion_limit = 10  # Prevent token explosion

4 Tools, Each with Hard Limits

@tool
def search_logs(service: str, keyword: str = None, level: str = None,
                start: str = None, end: str = None, limit: int = 100) -> list:
    """Search logs, hard limit ≤100 entries returned"""
    if limit > 100:
        limit = 100
    return center_client.search(...)

@tool
def count_logs(service: str, level: str, start: str, end: str) -> dict:
    """Count numbers, returns no content, only numbers"""
    return center_client.count(...)

@tool
def list_services() -> list:
    """All service names currently reporting logs"""
    return center_client.services()

@tool
def recent_snapshot(seconds: int = 30) -> list:
    """Logs from the last N seconds (max 300), to determine if errors are ongoing"""
    if seconds > 300:
        seconds = 300
    return center_client.recent(seconds)

Why does search_logs have a hard limit of 100 entries? This is an economic decision, not a technical one. Stuffing 1000 log entries into an LLM context could cost ¥0.5 per call. Using it 100 times a day would be ¥50. 100 entries are enough for the LLM to understand patterns and provide useful insights, while keeping token consumption under control.

litellm Abstracts Away Multi-LLM Differences

from litellm import completion

response = completion(
    model="qwen-plus",  # One-line switch: openai/gpt-4o, anthropic/claude-3.5, deepseek/deepseek-chat
    messages=[...],
    tools=[...],
)

Comparison with Mainstream Market Options

How to Use

After all that talk, how do you actually use it? Very simple, zero intrusion.

<dependency>
  <groupId>io.github.yjn-pj</groupId>
  <artifactId>lightlog-collector-spring-boot-starter</artifactId>
  <version>1.2.5</version>
</dependency>

lightlog:
  enabled: true
  server-url: http://lightlog-center:8088
  service-name: my-service

One pom snippet + 4 lines of yml, without changing a single line of Java code, and it's done. The rest is ➡️Deployment