How to set up dbt Core

Have more questions? Submit a request

(Optional) Create your own S3 bucket

πŸ’ͺ Did you know? We recommend uploading the required files to the same S3 bucket as Atlan, to avoid access issues. Reach out to your Data Success Manager to get the details of your Atlan bucket.

If you opt to create your own S3 bucket, you will need to do these additional steps:

Add IAM policy

Add the following IAM policy to the default EC2 instance role used by the Atlan EKS cluster:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "VisualEditor0",
      "Effect": "Allow",
      "Action": [
        "s3:GetBucketLocation",
        "s3:ListBucket",
        "s3:GetObject"
      ],
      "Resource": [
        "arn:aws:s3:::<bucket-name>",
        "arn:aws:s3:::<bucket-name>/<prefix>/*"
      ]
    }
  ]
}
  • Replace <bucket-name> with the name of the bucket you are creating.
  • Replace <prefix> with the name of the prefix (directory) within that bucket where you will upload the files.

Create a cross-account bucket policy

To create a cross-account bucket policy, when you have deployed Atlan as Single Tenant SaaS:

  1. Raise a support ticket to get the ARN of the Node Instance Role for your Atlan EKS cluster.
  2. Create a new policy to allow access by this ARN:
    {
      "Version": "2012-10-17",
      "Statement": [
        {
          "Sid": "VisualEditor0",
          "Effect": "Allow",
          "Principal": {
            "AWS": "<role-arn>"
          },
          "Action": [
            "s3:GetBucketLocation",
            "s3:ListBucket",
            "s3:GetObject"
          ],
          "Resource": [
            "arn:aws:s3:::<bucket-name>",
            "arn:aws:s3:::<bucket-name>/<prefix>/*"
          ]
        }
      ]
    }
    • Replace <role-arn> with the role ARN of Atlan's node instance role.
    • Replace <bucket-name> with the name of the bucket you are creating.
    • Replace <prefix> with the name of the prefix (directory) within that bucket where you will upload the files.

Structure the bucket

Multiple projects

Atlan supports extracting dbt metadata from multiple dbt projects. You need to use one of the following structures:

Environment-inclusive Without an environment
main-prefix
  • environment1
    • project1
  • environment2
    • project2
    • project3
  • environment3
    • project4
    • project5
main-prefix
  • project1
  • project2
  • project3
  • project4
  • project5

Both examples will be processed as five different dbt projects. The base folder name (for example, project2) will be stored as Project Name in the dbt metadata.

Single project

For a single dbt project you can directly upload files in the main S3 prefix or inside another folder that has the dbt project name.

Project-inclusive Without a project
main-prefix
  • project1
main-prefix

Upload project files

Upload the following files from the target directory of the dbt project into one of the bucket structures outlined above:

  • catalog.json
  • manifest.json
  • (Optional) sources.json β€” this will enrich source freshness data in Atlan, and is generated by running dbt source freshness
  • (Optional) run_results.json

Related articles

Was this article helpful?
0 out of 0 found this helpful