dbt Cloud is a SaaS product offered by dbt that allows you to build, run, document, and manage your dbt projects. Instead of using a custom orchestration tool to compile dbt models, you can leverage the infrastructure provided by dbt.
There are two deployment options available in dbt Cloud: Single Tenant and Multi Tenant.
You can learn more about them in the following links: Single Tenant and Multi Tenant.
Multi-Tenant deployments are the default option and are managed by dbt as a SaaS solution.
Single Tenant deployments, on the other hand, provide dedicated infrastructure in a virtual private cloud (VPC) environment. Although dbt Labs still manages and maintains them, they offer dedicated resources within a VPC. This is achieved by creating the necessary infrastructure using a reusable Infrastructure as Code (IaC) deployment built with Terraform.
One of the exciting features of dbt Cloud is the ability to use APIs to fetch metadata.
dbt Cloud provides two types of APIs for this purpose:
- dbt Cloud Administrative API: This is a REST API that is available for both single-tenant and multi-tenant deployments.
- dbt Metadata API: This is a GraphQL API that is only available for Multi-Tenant deployments.
We utilize these APIs to retrieve metadata about all the assets and process it to generate dbt model sources. This information is then used in Atlan to handle dbt assets and lineage.
When integrating with dbt Cloud in Atlan, we require the dbt Cloud API token and the deployment type.
We use this information to make API requests accordingly, as the availability of APIs and extraction patterns differ between Single Tenant and Multi-Tenant deployments.
For dbt Cloud (Single Tenant):
- Accounts, Projects, Environments, Jobs, and Runs are extracted using the cloud admin API.
- Models, Sources, Metrics, and Seeds are extracted from the
manifest.jsonfile, which is fetched as an artifact of the latest job run using this cloud API call.
For dbt Cloud (Multi Tenant):
- Accounts, Projects, Environments, Jobs, and Runs are extracted using the cloud admin API.
- Models, Sources, Metrics, and Seeds are extracted using the metadata API.
