dbt Cloud: Production Gatekeeper Demo

Hey folks, this is song speaking. And today I'm going to give a demo of gate keeping your production data, and let's take, let's take a step back to figure out what problem I'm solving for in the first place.

Um, when it comes to production data from your D B T jobs, uh, when you, when things go well, it's, it's pretty easygoing to, to navigate, but when things go wrong, it's typically too late because that bad data is already living and breathing in production.

But what if there was a better way where when problems happen, not if, uh, when they happen, uh, they only happen at a pre-prod staging layer, for example.

And so that staging layer can crash and burn all we want, but production data is never touched. The only time production data is touched when we know everything, uh, with good certainty is working in a staging layer.

And we essentially copy over that data into production and move on with our lives. A lot of this was inspired by Ben Stan's latest blog around here of essentially illustrating that same point of everything is written to a staging schema.

And then when things are green checkmark, all the way we copy that over into production, what's even more inspiring is boxy Sean or Sean, uh, already detailed this kind of mechanism back in July, 2021.

And I'm going to replicate these exact steps in a living and breathing DVD cloud job today. And you'll have some code, uh, to get started as well.

So step one, I'm going to create steps that look like this, and I simulated an error over here. So overall step by step, I run a macro called zero copy clone pre-prod, which essentially does the below.

It drops the tables and views in my target schema, which is staging. And it clones it from the production data in scope.

Now, mind you, this demo only works in the context of schema. Step two is I'm gonna simulate these errors. I add a worn error here.

So anything that's a warning is considered an error and failure and it skips everything subsequently. And so over here, I did that down here.

Typically this is a warning. I turned it into a failure and it tells me these are all the things that went wrong and everything that skipped as a result of that.

And then the last step over here is not run at all. Now let's evolve this even further and say, Hey, I actually want to see what this looks like when everything is going well, that's where we go over here.

I'm gonna update my settings. I'm gonna go over here and edit it. And I'm gonna remove this extra flag. I'm gonna click save.

I'm gonna go back to this job. I'm gonna run it and let's see what happens from here. It should be going through those same exact steps, but this time green check Markel around real quick.

I wanna show you the logs of this step, just so you can see how everything is defaulting to this sun staging schema, which is what I want.

Okay? Because we wanna make sure everything in our latest code changes are running against the staging layer before anything touches prod.

And I'm going to show you even further. Hey, how exactly do you configure this at the environment level? I'm gonna go over here.

I'm gonna go to my production gatekeeper environment. I'm gonna click on settings. And over here, I'm using 1.3. I'm running against this custom branch.

I'm defaulting to this database and I'm making sure any job that's run is against my staging layer. I'm gonna go back here.

I'm gonna click on this job again. I'm gonna click on what's running and fingers crossed. Everything works as expected. Okay?

Now that everything is run just to go through this one more time. I clone from my production schema into my staging schema, run all my latest code changes against the staging schema.

Everything's looking good as expected and then clone it back from my staging schema to my production schema. And now I know with great certainty, that production is exactly what I expect.

And next time things go wrong in a production job like this. I know my errors are being prevented from actually entering into production and a lot less headaches occur.

All right. That's it. See you.