The Agentic Data Stack: Speed and Simplicity for AI That Scales

Session Overview

AI agents can produce incorrect answers behind responses that appear completely normal, making failures difficult to detect with traditional observability tools. This session explores how organizations can build an Observe → Evaluate → Improve operational loop using Langfuse, the open-source AI observability platform.

From a practitioner's perspective, the session will cover tracing and cost tracking, online and offline evaluation with LLM-as-a-judge, CI/CD-integrated regression gates, and the path toward AI agents that can eventually run the improvement loop themselves. It will also examine the role ClickHouse plays in supporting AI observability workloads reliably and efficiently at scale.

Key Takeaways

• Why traditional observability is not enough for production AI agents
• How to implement tracing, cost tracking, and evaluation with Langfuse
• Using LLM-as-a-judge for online and offline quality evaluation
• Building CI/CD regression gates to prevent quality degradation before deployment
• The path toward self-improving AI agents and the role of ClickHouse at scale