Don't ship what you can't measure: Evaluation-driven development for AI analytics agents
Data Culture couldn't answer a client's question about agent accuracy, so it built a three-step evaluation framework that pushed one deployment from 0 percent to 94.7 percent.
Overview
Introduces "evaluation-driven development" for AI analytics agents: define correctness standards before building, measure each iteration, deploy only when evidence supports it. Covers test question sets from real scenarios, automated evaluation pipelines, and tracking accuracy gains, grounded in Snowflake Cortex Analyst deployments in healthcare and e-commerce.
What you'll take away
- 01
"Don't ship what you can't measure": Data Culture couldn't answer a basic question, what's this agent's accuracy, during its first client deployment, which became the trigger for building a formal evaluation framework.
- 02
A naive exact-match comparison marked correct answers as wrong because of cosmetic mismatches, like a differently named column, so the team built a four-tier comparison cascade before scoring anything as a failure.
- 03
On one deployment, accuracy climbed from 0% with no semantic context to 94.7% after layering in modeled dbt tables, column descriptions, business rules, and verified queries, with the same model used at every step.
Session speaker
Opeyemi Fabiyi
Data Culture