From 72 minutes to 5: how one environment variable powers two dbt pipelines
One health network needed the same business logic to answer in five minutes and still hold up to a full nightly audit, and it built both out of a single dbt model.
Overview
Demonstrates using a single environment variable to run two pipelines from one codebase: a 'live' mode filtering active records every five minutes for real-time dashboards, and a 'historical' mode for analytical work. Implemented at a health network managing dashboards across nine hospitals, achieving a 93.6% runtime reduction and 95%+ less data processed per run. Covers code examples, DAG propagation, and platform job configuration.
What you'll take away
- 01
One env_var('ENV_NRT') check lets a single model run two execution modes, active records on a five-minute table build, full history as an overnight incremental, from the same codebase.
- 02
The results: a 93.6% runtime reduction, from 4,340 seconds down to 278, and more than 99% less data processed per run in near-real-time mode, cutting refresh cadence from 72 minutes to 5.
- 03
Inner-join propagation is the cheapest win in the pattern: design the upstream anchor model correctly, and the narrower scope flows through the DAG for free.
Session speakers
Maksim Romanchuk
WMCHealthPaul Pedrozo
Westchester Medical Center