Troubleshooting dbt connectivity

What are the known limitations of the dbt connector?

Following are the known limitations:

  • For dbt Core, Atlan currently only processes the status of dbt tests — passed, failed, and warning — but not the job name and execution time. However, this information is available for dbt Cloud. You can hover over the test Status in the asset sidebar to view this information.
  • Column-level lineage for dbt models is currently unavailable at source. However, Atlan uses a custom name-match algorithm to generate column-level lineage for dbt models. Note that this may lead to certain discrepancies in lineage.
  • Ephemeral dbt models do not materialize tables at source. Hence, these will neither create lineage for the materialized layer nor publish dbt model columns in Atlan. This is an expected behavior of ephemeral dbt models.
  • Lineage between dbt sources and models is not supported. If a dbt model does not have any explicit dependency on dbt sources or other dbt models, Atlan will parse the compiled SQL of the dbt model to generate lineage to SQL warehouse tables at source.
  • For dbt models that have dependencies on both native dbt assets as well as SQL assets, Atlan currently only supports generating upstream lineage to native dbt assets, such as dbt sources and models. In such instances, upstream lineage to SQL sources will be missing. This is because Atlan currently only parses the compiled SQL when a model does not have any dependencies on native dbt assets.

Does Atlan support data types for dbt model columns?

Yes, Atlan supports column data types for dbt models:

  • dbt Cloud — no additional configuration required
  • dbt Core — upload the catalog.json file to extract column data types

Why are columns for dbt models missing?

If you've crawled your dbt models but columns are missing:

  • Ensure that the columns are defined in the columns attribute of the model's .yml definition in the manifest.json file.
  • Check if the dbt model is materializing a table or view asset, without which columns for a dbt model will not show up.

Is dbt source metadata available for materialized columns?

If metadata enrichment is enabled for assets that dbt materializes, only table assets will be updated with the materialized by dbt source or model metadata. Materialized columns are only updated with dbt model metadata — dbt source metadata is currently not supported for column assets. 

Can I map the Atlan GitHub action to multiple dbt projects?

Yes, you can configure the Atlan GitHub action for multiple dbt projects. 

Does Atlan support syncing terms from dbt?

Yes, you can update terms from dbt to Atlan. For more details, refer to our developer documentation.

Why are some dbt tests missing?

Atlan does not support crawling dbt tests with an auto-generated unique_id that exceeds the character limit of 32,000 characters. If you want to catalog such dbt tests in Atlan, you will need to define a custom name for your dbt tests within the character limit.

Why is there a discrepancy in dbt test count between dbt and Atlan?

dbt fetches models and tests for your entire dbt environment whereas Atlan fetches models and tests for each dbt job using the Discovery API, allowing for job-level filtering. The discrepancy in resource count may result from the difference in querying methods. While dbt displays the resource count across your entire dbt environment, Atlan fetches models and tests based on jobs.

Related articles

Was this article helpful?
0 out of 0 found this helpful