How to crawl Fivetran

Once you have configured the Fivetran user permissions, you can establish a connection between Atlan and Fivetran.

To enrich Atlan with metadata from Fivetran, review the order of operations and then complete the following steps.

Select the source

To select Fivetran as your source:

  1. In the top right of any screen, navigate to New and then click New Workflow.
  2. From the list of packages, select Fivetran Enrichment and then click Setup Workflow.

Provide credentials

To enter your Fivetran credentials:

  1. For API Key, enter the API key you generated.
  2. For API Secret, enter the API secret you generated.
  3. Click the Test Authentication button. This confirms you have entered a valid API key and secret pair to connect to Fivetran.
  4. Once authentication is successful, in the Preflight checks box, click the Check button. This confirms that your API key has both the Account Reviewer role and access to Fivetran's metadata API.
  5. When successful, at the bottom of the screen, click Next.

Configure the connection

To complete the Fivetran connection configuration:

  1. Provide a Connection Name that represents your source environment. For example, you might use values like production, development, gold, or analytics.
  2. (Optional) To change the users able to manage this connection, change the users or groups listed under Connection Admins.
    ๐Ÿšจ Careful! If you do not specify any user or group, nobody will be able to manage the connection โ€” not even admins.
  3. Navigate to the bottom of the screen and click Next to proceed.

Configure the enrichment

Before running the Fivetran enrichment, you can further configure it.

On the Metadata page, you can override the defaults for any of these options:

  • For Advanced Config, keep Default for the default configuration or click Custom to configure the enrichment:
    • For Publish process announcements, click Yes to allow Atlan to publish process announcements for warnings generated within Fivetran or click No to disable them. 

Run the enrichment

You can now enrich Atlan with Fivetran metadata.

  • At the bottom of the screen, click Run to run the enrichment once, immediately.
  • Click Schedule & Run to schedule the enrichment to run hourly, daily, weekly, or monthly.

Once the enrichment has completed running, you will see lineage extended upstream from your data platform, warehouse, or lake! ๐ŸŽ‰

Related articles

Was this article helpful?
1 out of 1 found this helpful