Troubleshooting dbt connectivity

What are the known limitations of the dbt connector?

Following are the known limitations:

• For dbt Core, Atlan currently only processes the status of dbt tests — passed, failed, and warning — but not the job name and execution time. However, this information is available for dbt Cloud. You can hover over the test Status in the asset sidebar to view this information.
• Column-level lineage for dbt models is currently unavailable at source. However, Atlan applies a custom SQL-parsing algorithm to the materialized SQL tables to generate column-level lineage for dbt models. Note that due to the limitations of SQL parsing, Atlan does not guarantee generating lineage for all columns.
• Ephemeral dbt models do not materialize tables at source. Hence, these will neither create lineage for the materialized layer nor publish dbt model columns in Atlan. This is an expected behavior of ephemeral dbt models.
• A dbt source can include multiple SQL warehouse tables. Lineage between dbt sources and models is not supported. Instead, Atlan builds lineage between dbt models and SQL tables when dbt models have an explicit dependency on specific SQL tables of a dbt source.

Why are my dbt Cloud models not showing up in Atlan after setup?

Atlan only crawls dbt assets that are in the “applied” (built) state in dbt Cloud. Models must be part of a successful run to be picked up during crawling; models that are only defined in your project files but haven’t been executed won’t be included. For more information about project state, see Project states in dbt Cloud.

Does Atlan support data types for dbt model columns?

Yes, Atlan supports column data types for dbt models:

• dbt Cloud — ensure that there is at least one job run with the Generate docs on run option enabled for every environment in which dbt models are executed.
• dbt Core — upload the manifest.json and catalog.json files generated by the dbt docs generate command for every dbt Core project. Refer to dbt Core documentation to learn how to structure the bucket while uploading your files.

Why are columns for dbt models missing?

If you've crawled your dbt models but columns are missing:

• Ensure that the columns are defined in the columns attribute of the model's .yml definition in the manifest.json file.
• Check if the dbt model is materializing a table or view asset, without which columns for a dbt model will not show up.

Is dbt source metadata available for materialized columns?

If metadata enrichment is enabled for assets that dbt materializes, only table assets will be updated with the materialized by dbt source or model metadata. Materialized columns are only updated with dbt model metadata — dbt source metadata is currently not supported for column assets. 

Can I map the Atlan GitHub action to multiple dbt projects?

Yes, you can configure the Atlan GitHub action for multiple dbt projects. 

Does Atlan support syncing terms from dbt?

Yes, you can update terms from dbt to Atlan. For more details, refer to our developer documentation.

Why are some dbt tests missing?

Atlan does not support crawling dbt tests with an auto-generated unique_id that exceeds the character limit of 32,000 characters. If you want to catalog such dbt tests in Atlan, you will need to define a custom name for your dbt tests within the character limit.

Why is there a discrepancy in dbt test count between dbt and Atlan?

Atlan fetches dbt models, sources, and tests from the applied state of each dbt environment. This is the most recent project state for each environment. The asset count on Atlan may differ from what’s present in the manifest files for individual job runs. Refer to dbt documentation to learn more about dbt project states.

Why does Atlan link the dbt model description to my SQL source table?

If a materialized table is linked to multiple dbt assets, Atlan will apply the description from the linked dbt model to the materialized asset. In case this is unavailable, Atlan will then apply the description from the linked dbt source to the asset.