How Sweetwater catches broken dbt pipelines before anyone notices, with Datadog

Daniel Gonzalez and Derek Andres, both analytics engineer tech leads at Sweetwater Sound, the largest online music gear retailer in the country, walked through how their team built a proactive data observability practice with Datadog as it scaled its dbt-based data platform out of Fort Wayne, Indiana.

Sweetwater has grown rapidly since 2020, expanding into new lines of business including its Gear Exchange and marketplaces on Amazon, Reverb, and eBay, while running a 600-plus person sales engineering team. One advantage the data team leans on: all of Sweetwater's developers are in-house, so when a source system needs a change, the data team can walk over and talk to the person who built it instead of coordinating across vendors.

The company's data stack went through several distinct eras. In the 2010s, Domo was introduced as a broadly accessible, drag-and-drop dashboarding tool that was treated like a data warehouse without really being one. By 2020, rapid growth had produced thousands of reports, duplicated transformations, and no formal governance.

In 2023, Sweetwater began migrating to a scalable, governed data infrastructure built on dbt, a project that touched 26 departments and engaged more than 100 stakeholders across business intelligence, spoke developers, data engineering, data science, and department subject matter experts. By 2025, Domo was fully deprecated and the new stack fully integrated. The dbt project itself grew quickly on a set of core ingestion macros, then steadily as the team tackled harder problems, and now carries more than 11,000 column descriptions.

Observability entered the picture as the data team's relationship with IT matured. The team had originally been siloed from the rest of IT, but as it integrated more closely, the need for cross-system observability became obvious, especially since DBAs and data engineers elsewhere in the org were already using Datadog to monitor source systems. Extending that same tool upstream into the data team's own pipelines was a natural fit rather than a new tool to sell internally.

The team's clearest example of that value: a user reported that the hourly sales dashboard was showing invoices with no attributed sales channel. Records were still trickling into the source tables, so the data looked fresh, and the issue kept recurring on a fairly stable weekly cadence. Using dashboards the data engineering team already had for Kafka inserts by topic, Gonzalez and Andres traced a spike in transactions back to one specific websql source, and from there found a legacy script that had been incorrectly scheduled.

That let them get the script rescheduled and start a conversation about eventually moving that workload into the cloud warehouse. A related challenge came from compute management itself: as their DAG expanded and batch frequency needs grew, they risked job timeouts when the warehouse ran hot, and Datadog's native Google Cloud integrations and prebuilt metrics helped them understand and smooth out that risk.

Sweetwater's current monitoring setup includes monitors configured automatically across entire schemas and databases, row count monitors that adjust to each table's expected volume, freshness monitors tuned to tolerate planned nightly downtime, customizable alert and recovery messaging routed to different targets for different teams, and machine-learning-driven alerting thresholds with a simple annotation framework for manual correction.

Looking ahead, the team is expanding into lineage and query monitoring that extends visibility beyond dbt itself, using PupCLI and the Datadog MCP server to make observability accessible from the terminal or an AI assistant, and adopting CI/CD observability features that automate impact analysis and data profiling for changed models directly in their git provider.

Watch every keynote and breakout from dbt Summit 2026 on demand. Watch dbt Summit 2026 on demand →