Fresh Rebuilds Tutorial
3 mins
Transcript
Show Transcript
Hey folks, this is sung speaking here today. I'm going to take you through a tutorial on how to do fresh rebuilts within DBT cloud.
And so let's take a step back. First of what problem we're trying to solve in the first place. It's, let's say you have an extract and load process that typically, you know, refreshes, you know, a hundred sources within your DPT project.
However, just because, you know, extract and load process finish, it doesn't necessarily add newer data or fresher data to the mix.
And so you could be running, you know, a hundred DBT models downstream in a wasteful fashion, right? Because your source data didn't change.
That also means your downstream data doesn't change either. Um, even if you transform it again and again and again. So at that in mind, what this solves for is, Hey, out of those hundred sources, let's say two out of the hundred sources actually have fresher data with that in mind, let's run all the downstream nodes, uh, of those two fresher sources and then ignore the rest.
Alright, so with that in mind, I just have a very simple example right here. I have a staging table where I kind of, you know, mimic a constantly updating source, uh, kind of fresh data node.
And so with that in mind, all I'm doing here, the selecting from a source, and then right now I'm appending fresher sources.
The key thing to note is line 31, where I'm adding the current timestamp each time. So each time I'm running this model, it should have fresher rows each and every time.
And then I define that demo source over here, where I simply name it, search Fisher demo, and then add a simple freshness test over here.
And then in addition to that, I add a placeholder descendant model of that source fresher note, um, in order to make sure that that is run downstream every time this is fresh.
And so from there, I actually set up a job in this case, I'm just naming it, source fresher job a I make sure to hit the DVD version 1.1, um, if that's not already configured within your environment, and then I already make sure to defer to self, right, because we want to check upon itself based off the previous source, fresh and stop happening over here.
And then I run the artificial kind of source fresh, a node that I just illustrated within the IDE. And then this is the syntax we should care about.
It's a source status, colon fresher plus. And so what this means is I'll test all my sources and run and test any of the downstream nodes as a result of this.
And then I turn off the schedule and that's it. So I'm going to cancel it to this. I'm going to go to source jaw fresher a I'm going to run it now.
And you'll notice here, I've run it a couple of times and what's going to happen here is that one it's going to queue up and then two, it's going to run the steps and make sure, Hey, let's check.
The previous time we ran source freshness, let's run source freshens again, during this current job and compare the max load to that dates between the previous state and the current state.
And if the current state max load of that date is greater, then we run the downstream steps. Okay. So for the purposes of this demo, I'm going to wait for that to finish And after this is done, you can see over here, we ran source freshness.
Again, that differs to the previous time we ran source freshness. We go over here and we notice, Hey, it only ran the descendant model and scope and ignore the rest.
And that's it.
Transcript
Show Transcript
Hey folks, this is sung speaking here today. I'm going to take you through a tutorial on how to do fresh rebuilts within DBT cloud.
And so let's take a step back. First of what problem we're trying to solve in the first place. It's, let's say you have an extract and load process that typically, you know, refreshes, you know, a hundred sources within your DPT project.
However, just because, you know, extract and load process finish, it doesn't necessarily add newer data or fresher data to the mix.
And so you could be running, you know, a hundred DBT models downstream in a wasteful fashion, right? Because your source data didn't change.
That also means your downstream data doesn't change either. Um, even if you transform it again and again and again. So at that in mind, what this solves for is, Hey, out of those hundred sources, let's say two out of the hundred sources actually have fresher data with that in mind, let's run all the downstream nodes, uh, of those two fresher sources and then ignore the rest.
Alright, so with that in mind, I just have a very simple example right here. I have a staging table where I kind of, you know, mimic a constantly updating source, uh, kind of fresh data node.
And so with that in mind, all I'm doing here, the selecting from a source, and then right now I'm appending fresher sources.
The key thing to note is line 31, where I'm adding the current timestamp each time. So each time I'm running this model, it should have fresher rows each and every time.
And then I define that demo source over here, where I simply name it, search Fisher demo, and then add a simple freshness test over here.
And then in addition to that, I add a placeholder descendant model of that source fresher note, um, in order to make sure that that is run downstream every time this is fresh.
And so from there, I actually set up a job in this case, I'm just naming it, source fresher job a I make sure to hit the DVD version 1.1, um, if that's not already configured within your environment, and then I already make sure to defer to self, right, because we want to check upon itself based off the previous source, fresh and stop happening over here.
And then I run the artificial kind of source fresh, a node that I just illustrated within the IDE. And then this is the syntax we should care about.
It's a source status, colon fresher plus. And so what this means is I'll test all my sources and run and test any of the downstream nodes as a result of this.
And then I turn off the schedule and that's it. So I'm going to cancel it to this. I'm going to go to source jaw fresher a I'm going to run it now.
And you'll notice here, I've run it a couple of times and what's going to happen here is that one it's going to queue up and then two, it's going to run the steps and make sure, Hey, let's check.
The previous time we ran source freshness, let's run source freshens again, during this current job and compare the max load to that dates between the previous state and the current state.
And if the current state max load of that date is greater, then we run the downstream steps. Okay. So for the purposes of this demo, I'm going to wait for that to finish And after this is done, you can see over here, we ran source freshness.
Again, that differs to the previous time we ran source freshness. We go over here and we notice, Hey, it only ran the descendant model and scope and ignore the rest.
And that's it.
More than 21 million people across 200,000 companies choose Loom
For Mac, Windows, iOS, and Android
My teammates and I love using Loom! It has saved us hundreds of hours by creating informative video tutorials instead of long emails or 1-on-1 trainings with customers.
Erica GoodellCustomer Success, Pearson
Loom creates an ongoing visual and audible experience across our business and enables our employees to feel part of a unified culture and company.
Tyson QuickCEO, Postclick
My new daily email habit. Begin writing an email. Get to the second paragraph and think 'what a time suck.' Record a Loom instead. Feel like 😎.
Kieran FlanaganVP of Marketing, HubSpot
Loom amplifies my communication with the team like nothing else has. It's a communication tool that should be in every executive's toolbox.
David OkuinevCo-CEO, Typeform
My teammates and I love using Loom! It has saved us hundreds of hours by creating informative video tutorials instead of long emails or 1-on-1 trainings with customers.
Erica GoodellCustomer Success, Pearson
Loom creates an ongoing visual and audible experience across our business and enables our employees to feel part of a unified culture and company.
Tyson QuickCEO, Postclick