How to set up dbt Core

Option 1: Use the Atlan S3 bucket

To avoid access issues, we recommend uploading the required files to the same S3 bucket as Atlan. Raise a support request to get the details of your Atlan bucket and include the ARN value of the IAM user or IAM role we can provision access to.

If you instead opt to use your own S3 bucket, you will need to complete the following steps:

Option 2: Use your own S3 bucket

You'll first need to create a cross-account bucket policy giving Atlan's IAM role access to your bucket 

1. Raise a support ticket to get the ARN of the Node Instance Role for your Atlan EKS cluster.
2. Create a new policy to allow access by this ARN:

   {
     "Version": "2012-10-17",
     "Statement": [
       {
         "Sid": "VisualEditor0",
         "Effect": "Allow",
         "Principal": {
           "AWS": "<role-arn>"
         },
         "Action": [
           "s3:GetBucketLocation",
           "s3:ListBucket",
           "s3:GetObject"
         ],
         "Resource": [
           "arn:aws:s3:::<bucket-name>",
           "arn:aws:s3:::<bucket-name>/<prefix>/*"
         ]
       }
     ]
   }

   • Replace <role-arn> with the role ARN of Atlan's node instance role.
   • Replace <bucket-name> with the name of the bucket you are creating.
   • Replace <prefix> with the name of the prefix (directory) within that bucket where you will upload the files.

3. Once the new policy has been set up, please notify the support team. Your request should include the S3 bucket name and prefix. This should be done prior to setting up the workflow so that we can create and attach an IAM policy for your bucket to Atlan's IAM role.

Structure the bucket

Multiple projects

Atlan supports extracting dbt metadata from multiple dbt projects. You need to use one of the following structures:

Both examples will be processed as five different dbt projects. The base folder name (for example, project2) will be stored as Project Name in the dbt metadata.

Single project

For a single dbt project you can directly upload files in the main S3 prefix or inside another folder that has the dbt project name.

Upload project files

Upload the following files from the target directory of the dbt project into one of the bucket structures outlined above:

• manifest.json, which you can generate by running:

  dbt compile --full-refresh

• (Optional) sources.json, which will enrich source freshness data in Atlan. You can generate it by running:

  dbt source freshness

• (Optional) run_results.json, which you can generate by running:

  dbt test

Set up dbt Semantic Layer

To use the dbt Semantic Layer:

1. Install the dbt_metrics package into each dbt project you want to expose through the semantic layer.
2. Define the metrics you want to expose in dbt. See dbt's Defining a metric article for more details.