dbt Monorepo Workflow: Getting Started

Hey folks, this is UNG speaking and I'm about to give a demo of the monorepo workflow. And let's take a step back to figure out why, uh, this matters in the first place.

Um, when I usually onboard, uh, new customers or just get people asking me questions of, hey song, how do I get started if I'm brand new to D B T?

And just want kind of a coherent reference point to understand how I get my teams to collaborate together without stepping on each other.

And how do we promote our code across environments, um, so that if and when things go wrong, there's a clear kind of breaking point to go like, oh, I know when to undo something.

I know when something's being emerged into prod. And there's this general peace of mind that things are flowing exactly as expected when you're making changes to your DPT project.

And so, um, I'm gonna demo a little bit how, um, this workflow will look and feel in concept by, you know, talking a little bit into actually showing you the points and clicks and what the UI and UX feels like in DPT Cloud.

And then lastly, I'll give you some quick tips. Um, it won't be exhaustive, but it will be enough to kind of point you in the right direction of, Hey, how do I make sure all the little configurations within DPT Cloud, you know, in GitHub are doing exactly what I want, you know, to essentially replicate

what I'm doing right now. All right, so let's get started. So step one is understanding conceptually how this all works.

Um, so if you're new to D B T, um, this is just assuming that you're making net new models, all right?

You're not migrating in anything over.

Okay? And so step one is to establish some teams. Let's just say we have like team A and team B, and, um, we have something called a QA project and a prod project.

So the main purpose of DT Cloud, your QA proj or your team project, something that looks, you know, like this in the upper right hand corner, this would be like team A, QA and prod.

The main reason we have that is one, you just have gated controls over who in D cloud is allowed to read and write what.

And so think like role-based access controls, things along those lines. And the kind of main reference point here is just making sure that, you know, out of, let's say you have a, a team of 10 users, um, you know, four of them belong, you know, five of them belong to team A, five of them belong to team

B, and then maybe one or two people have admin rights to access the QA project and the prod project.

And the main reasons for that is to, you know, ensure, you know, people aren't hacking and slashing away and changing settings willy-nilly throughout the experience.

Um, and so that's some groundwork there. Then you have some warehouses that could be in the context of Snowflake or Databricks, what have you, but essentially they're referring to, you know, compute.

Um, so that you know, hey, team A is only touching, you know, team a's you know, warehouse to make sure you attribute and map costs to.

Um, same thing for QA and prod dev is usually like in, in the snowflake context, an extra small or small qa.

There may be like a medium, medium or large. And same thing for prd, just to speed things up and replicate what things actually look and feel like in production.

Um, over here we have a monorepo. Again, this is in the context of a monorepo.

If you have questions about multi repo that's out of scope for this conversation. Um, and so yeah, that's some of the groundwork for what you're seeing in this slide.

And so I, I think it helps just to talk through a story here. So let's say I am an analyst on, on team A.

Okay? Step one, I've been tasked with building a net new model or adjusting something. And so step one, I'm going to create a feature branch that's like feature A, and then this is assuming that I also have other teams, let's say across teams, B, C, D, et cetera, that are also building features.

And what's nice about this is since we're all pulling from the same QA branch as kind of a reference starting point, uh, we all know, uh, that there's a consistent baseline.

So there's no second guessing of, hey, uh, are they starting from the same place as me?

And the, and the answer is, yes, they are. And so feature a I'm owning that. And then once that's complete, um, I merge it into qa, make sure that DV cloud kicks off some slim CI jobs, um, to make sure all my net new changes are working exactly as expected.

So, you know, the TLDR is if I see good green check marks, things are good to go, and then, um, I wait for my other teammates to finish their feature requests.

And once you bundled up a bunch of those into qa, uh, then typically on a weekly cadence, think like on Monday morning, um, someone who's an admin on QA initiates a pull request for all these bulk, uh, merges into the prod branch.

And then from there, there's one last sanity check just to make sure all those, um, feature requests, you know, merchant to one in the aggregate, uh, don't have any kind of, you know, unexpected consequences.

And once you have that one green check mark over here, then you merchant into prod, and the next time the, you know, the daily, uh, production job runs, uh, for example, then everything's good to go.

And so that's the general flow of everything and how it looks like. So in summary, you know, step one, uh, make a feature request, wait for a bunch of other feature requests to be bundled up on a Monday morning.

The QA branch makes a poor request into the prod branch. Uh, step three, make sure that, uh, is passing as expected and go from there.

And for those of you that are more, you know, textual learners, uh, I recommend just pausing this and looking at this step by step highlight so you have just a clear coherency as to what exactly is going on when.

All right, so now the next thing is to go through what this actually looks and feels like. So I'm gonna do actually a couple things here.

I'm going to move this to the side by side panel just so you understand how this visual maps to what's going on in D P T Cloud.

Okay, so step one, I'm in team A, I am in my second model. I was tasked to, hey song, actually we need to update this filter to six for whatever reason.

Okay? We wanna isolate specific users. So I'm making this one tiny change and then from there it is going to show me that I made these changes and then it's gonna auto update, uh, my gi um, experience to say, Hey song, we noticed these changes, you need to commit and sync them.

So I'm gonna do this change filter for second model. Okay? From there, I'm gonna commit those changes and then I'm gonna initiate a PO quest to go simulate what feature A is doing on the image on the left

Designing.

Okay? Next step is to create a pull request. I'm gonna go ahead and create it. I'm gonna just, um, bulldoze through this just to show you the mechanics.

And from there you notice here I've been working on this branch for a little bit. Um, and so it has previous commits up there, but the main thing I wanna point you to is, you know, a couple of these green check marks, you're probably wondering, especially if this is your first time, what exactly is going

on? This is D cloud automatically running a uh, CI job or continuous integration job just to adjust all the net new changes are working exactly as expected.

Now you're probably wondering what exactly does that mean step by step, if we zoom in, it'll give you a hyperlink to the exact job run that's taking place.

And it's doing a couple things here.

Um, it is one comparing the changes from my feature branch here, which is cool feature team a dev. You're probably wondering how exactly did I create that?

That's when I did this, you know, picked a branch or you can make a net new branch. Um, and then from there it is running a couple steps and I implemented something called slim ci.

And you know, in summary of what that means is, let's say you have a hundred models, I made one tiny change.

You're probably wondering, do I have to run all 101 changes just to test this one tiny thing? And thank that the answer is no, because with some CI it knows, hey, uh, deputy cloud is like, Hey son, I already know that you're making this one any change and I'm gonna run that.

And all the downstream changes as a result of that. And so you are only getting the subset of things run, saving you time, money, and energy understanding what exactly has been changed.

And so from there, we see a couple things here, ignore this, uh, for now, but I ran a couple things and I'm gonna, you know, what's it called?

Uh, refresh this just so you see, you know, some clearer logs here.

Okay, over here, you know this, all the things that ran my second model that makes sense. You notice here, it throws it into a temporary PR schema just so you know where exactly your changes are within your respective database.

And then, yeah, everything looks good from there. So now I'm thinking to myself, great, so from here, this feature a and now I wanna merge that into the QA branch.

In this case that's QA demo. Okay, so here, I'm just gonna say looks good to me, and then I'm going to squash and merge.

And all that's doing is taking up all the previous, uh, commits and putting them into one just so it's like one bundled up change being merged into my QA branch.

Okay, cool. Now that that's merged, now you're probably wondering, hey, what's the step to make QA go into prod? There's a couple things you can do here.

You can either do it directly through GitHub or through the ide. It probably most sense makes most sense from a UX UI perspective just to do it in GitHub.

And so from here I'm probably wondering, okay, what's the next step from here? I'm gonna go pull request.

I'm going to do, let's see, new pull request. I'm going to change this to prod demo, and then I'm going to change this to QA demo, okay, now I'm going to create the pull request, demo merge workflow.

Okay? I'm gonna create this pull request, okay? And then the branch has conflicts that must be resolved. And then in this case, let's see here, this is happening on the fly, so let's see if I can fix it.

One conflict, you know what? I'm probably gonna do this and then Mark as resolved, you know, in a real life setting, you'd probably wanna have a conversation around this.

But for the purposes of this, I just wanna show you the mechanics. It's over here. You see something is activated, and you're probably wondering what exactly is going on over here.

This is where a job in production is being run. So I'm gonna go over here and it's gonna show me the exact job.

You notice how I'm in the production DVD cloud project. And then from there it's gonna run a couple steps.

And then once I see green check marks all around, similar to the experience that we showcase from featured qa, um, I should be good to go and I'll squash and merge similar to feature A into qa.

And the next time the production job runs, let's say, you know, tomorrow at 8:00 AM it'll include all the changes in scope and we're good to go.

And now, while this is running, there's a couple things here. It's like, look, what if anything goes wrong in any of these respective steps?

You have, you know, healthy levels of gatekeeping at the QA branch level and at the PR branch level. So let's say for example, we saw green check marks, but there was like some wacky variable that happened in the actual production job at 8:00 AM the next day, I could easily undo this, you know, um, I

should say revert this poor request and then go back to the previous date of code that we know worked for sure.

And so, at any point in time, you are second guessing your confidence as to when things work or you know, how you're gonna answer when things go wrong, how do you fix it?

Um, those are your answers. You have gatekeeping at the future branch level, the QA branch level, and the prod branch level to a reasonable amount.

We are not pulling out your hair wondering like, do I have to, you know, click 10 things and talk to 20 people before I can merge anything?

And thankfully the answer is no. You just have two to three touch points and you're good to go from there.

Okay? I see these green things I squash and merch from squash and merch, and that's it. You get to move on with your life.

All right? And so that was kind of the, the quick and easy of how this monorepo workflow kind looks and feels.

Now, you're probably wondering, Hey son, what are all the little configurations in D Cloud to make sure that, you know, I can replicate exactly what you're doing?

All right, so there's a couple things we need to do here. First I am, let's see here, I am in qa.

So let's start with team A. Uh, let's have a kind of coherent picture with all this. Okay? And then I have, um, let's see here.

Oh, a couple things. I need to set up my environment and make sure it's development. So I think it's, and mind you, I'm using an older version of DB t.

Um, but the things that you'd want to do here is this one, probably name it. Team A development, your team name plus development.

You pick your DBT version, most likely the latest one. So 1.4 in this case, default to a cust.

So this is unchecked. You'd want to check this and make sure it defaults to the custom branch because that should be your reference point.

Anytime, um, you're making a new pull request, okay, I'm gonna go back here. Um, the next thing to do is once this is set up is, um, also make sure your respective, um, profile settings are aligned to, and in this case I'm using Snowflake, um, to the right credentials in the schema.

But you know, we'll address that another time that's outta scope for this conversation. And then from there, when you click develop, it should make sure it references that as the, the, the new main branch, and then you can make features off of that.

And so in this case, when you saw, you know, cool feature team, a dev branch, um, that was based off the QA branch as its starting point, okay?

Okay, next step from there is how do I make sure the QA project is doing exactly what I wanted to?

A couple things. Let's go back to environments over here. And then you will have QA deployment. Mind you, this is deployment environment, not a development one.

And so I'm gonna name it qa, pick that I'm gonna run on a custom branch and make sure, you know, I'm picking the respective cooling connection and deployment credentials for this.

And in this case it'll be a QA schema. Okay? Next up from there is I'm gonna go to jobs. Okay, great <laugh>.

And this case, I'm going to show you two things. You're gonna have a compile only job. And the main purpose of this is to, um, make sure it's showing you all the code as a reference point so that slim CI understands the changes that are being made with each PO request that it's running.

So in this case, it's running on a schedule just to make sure it's always running the, the latest code changes.

So I'm gonna go back here, I'm gonna go to feature to qa. Yeah. And the main purpose for this job is to do, um, a couple things here.

One, you wanna make sure you're deferring to the compile only job, so understands the net new changes. These are really optional.

You don't need to run these if you don't. Um, 80% of the time people uncheck these. Uh, but the big thing you need to note here is this.

Mind you, this is, um, built before I implemented the the D P T build command, which does run and test in parallel.

But in this case, if you want to, you know, have cleaner logs to understand what's being run and tested, you can just do this and make sure, the key thing to notice is this syntax over here.

Um, understandable. This isn't something intuitive, you should note out the box. Um, but select state modified plus over here is the main thing you should care about to understand, um, that it's only running the net new changes in scope and unit is here for tests.

And this select store failure. So it also stores it in a, um, test schema so you understand what exactly the failed rows look like.

And over here I turn off the schedule over here I run floor quests and run only custom branch, um, because they're running in the QA branch, right?

Okay. And the next thing to note is, hey, what is production? Uh, what is a, a production supposed to look like?

So over here, very similar experience over here. I go back to my environments, I click on product deployment, I click on settings, and same premise over here, and make sure it's running on the prod demo branch as its main reference point.

All the respective deployment and development credentials since over here it's prod demo. So, all right. And then next from there, I'm gonna show you the jobs.

Same thing. There's a compile only job. You know, nothing too crazy here, but I'll just show you just for, you know, sanity's sake that is doing exactly the, like how I said it, <laugh> it, it does for the QA context and the copies and paste over into the, the prod context.

I run a schedule and make sure it's TV to compile. That's it. Go back to jobs. I go back to my QA to prod, pull request jobs, ci, and then over here, same thing.

The main things to notice I'm deferring to a job. In this case, I, I defer to the the daily job cuz that's the job that actually runs all the things in scope.

And so you notice here, Hey, how come I'm running compile only in QA versus running a comparing against a daily job that's actually running stuff in prod, um, compile.

It's because so many feature requests are being made. Um, you don't always wanna run all those changes all at once.

Okay? And when it comes to prod, you actually wanted to reference the things that actually ran. Um, because the main thing to note when it defers to a previous run state that it only defers to the previous successful run state.

And so when you're running your changes, it's um, it's reference point is a previous successful run. And that's important to note cuz we wanna make sure that, um, there's no kind of unexpected, you know, surprises as to to what's running versus not.

And then con continuous integration run on Poor Quest. Run on custom branch. Yes. And then that's it.

Some quick tips on some other things you should care about is like, hey, is there a reference point for like what a directory structure looks like?

You know, for things like team A, B, C, D include this link in the video, but essentially GitLab role models a lot of this for us where you have workspaces and then there's a workspace for finance marketing people and that's, you know, sys for team A, B, C, D.

Another thing is, hey, how do I make sure that team A is only touching the files and team A, subfolder versus team B?

That's where if using GitHub for example, use something called code owners. And this essentially says, Hey, if team A, you know, makes some changes in the team B subfolder, it will automatically notify the Team B owner to, and then you can have a conversation from there to say, Hey, team A, you're not

supposed to touch the subfolder.

You know, I think you're meant to put this in subfolder A and then you can move on from there. And now you're also wondering, hey, what's a good template for a pull request?

This one's from 2021, but I think it still holds true. Um, where you can literally just copy and paste what we wrote here and move on from there.

And so in summary, now you know the story of what it looks like to get started with DB T and the Monorepo workflow.

Your teammate, you make a feature branch, you work with other teammates in parallel, you all merge your bulk changes into qa.

QA takes those bulk changes, mix a PR into prod, green check marks all around, merge into prod profit. All right?

Um, lemme know if you have any questions or comments and uh, we can go from there. All right. Peace.