Building Observability on a Real-World AI Platform

Session Overview

AI agents can return incorrect answers behind responses that appear completely normal, making failures difficult to detect with traditional observability tools. This session explores how to build an Observe → Evaluate → Improve operational loop using Langfuse, the open-source AI observability platform.

Drawing on real-world implementation experience, the session will cover tracing and cost monitoring, online and offline evaluation with LLM-as-a-judge, CI/CD-integrated regression gates, and the path toward AI agents that can eventually run their own continuous improvement loops. It will also examine how ClickHouse supports large-scale AI observability workloads with the speed, reliability, and scalability required in production environments.

Key Takeaways

• Why traditional observability is not enough for production AI agents
• How to build an Observe → Evaluate → Improve operational loop
• Implementing tracing, cost monitoring, and evaluation with Langfuse
• Using LLM-as-a-judge for online and offline quality evaluation
• Building CI/CD regression gates to prevent quality degradation before deployment
• The path toward self-improving AI agents supported by ClickHouse at scale