How to crawl Fivetran

Updated May 24, 2023 15:49

Have more questions? Submit a request

Once you have configured the Fivetran user permissions, you can establish a connection between Atlan and Fivetran.

To enrich Atlan with metadata from Fivetran, complete the following steps.

Select the source

To select Fivetran as your source:

In the top right of any screen, navigate to New and then click New Workflow.
From the list of packages, select Fivetran Enrichment and then click Setup Workflow.

To enter your Fivetran credentials:

For API Key, enter the API key you generated.
For API Secret, enter the API secret you generated.
Click the Test Authentication button. This confirms you have entered a valid API key and secret pair to connect to Fivetran.
Once authentication is successful, in the Preflight checks box, click the Check button. This confirms that your API key has both the Account Reviewer role and access to Fivetran's metadata API.
When successful, at the bottom of the screen, click Next.

To complete the Fivetran connection configuration:

Provide a Connection Name that represents your source environment. For example, you might use values like production, development, gold, or analytics.
(Optional) To change the users able to manage this connection, change the users or groups listed under Connection Admins.
🚨 Careful! If you do not specify any user or group, nobody will be able to manage the connection — not even admins.
Navigate to the bottom of the screen and click Next to proceed.

Before running the Fivetran enrichment, you can further configure it.

On the Metadata page, you can override the defaults for any of these options:

For Advanced Config, keep Default for the default configuration or click Custom to configure the enrichment:
- For Publish process announcements, click Yes to allow Atlan to publish process announcements for warnings generated within Fivetran or click No to disable them.

You can now enrich Atlan with Fivetran metadata.

At the bottom of the screen, click Run to run the enrichment once, immediately.
Click Schedule & Run to schedule the enrichment to run hourly, daily, weekly, or monthly.

Once the enrichment has completed running, you will see lineage extended upstream from your data platform, warehouse, or lake! 🎉