LightLog Replaces the ELK Stack with PostgreSQL and Adds an AI Troubleshooting Agent
Teams running small-to-medium services often face a bad tradeoff: ELK is too heavy, Loki's full-text search is too weak, and SaaS log tools get expensive fast. LightLog shows that PostgreSQL, with the right schema and index choices, handles log workloads well enough to skip dedicated search engines entirely, while the built-in AI agent turns log diving from a grep slog into a conversation.
LightLog is a new open-source log platform built to sidestep the resource and ops burden of Elasticsearch. It stores logs in PostgreSQL using daily partitions, BRIN indexes for time-series data, and JSONB with GIN indexes for semi-structured fields, returning results from 30 million rows in 34 milliseconds. Storage is pluggable across SQLite, H2, and PG via an environment variable.
Integration requires adding one Maven dependency and four lines of YAML to a Spring Boot service; a custom Logback appender ships logs asynchronously through an in-memory queue to a write-ahead log on disk before pushing to the central HTTP API, so business threads never block on network calls. The WAL replays on restart, preventing log loss during outages.
An AI assistant built on LangGraph and litellm exposes four hard-limited tools—search, count, list services, and recent snapshot—so operators can ask questions like "what errors did ths-api have today?" and get a fingerprint analysis without burning tokens on unbounded context windows.
Hard-limiting the AI agent's search tool to 100 log entries is framed as an economic decision, not a technical one—stuffing 1,000 entries into an LLM context could cost ¥0.50 per call, and 100 entries are enough to spot patterns.
The WAL design is explicitly justified by user trust: once logs go missing, confidence in the system collapses, so durability is treated as a minimum promise rather than over-engineering.
BRIN indexes are a pragmatic fit for append-only log data because they store only block-level min/max values, shrinking index size to roughly 1/100th of a B-tree, though they trade off some scan precision.
Pluggable storage via a Spring factory pattern and environment variable means the same codebase runs with zero external dependencies in development (SQLite) and scales to PG in production without config rewrites.