How to set up Fivetran

Atlan currently supports fetching metadata from Fivetran using:

• Fivetran Platform Connector — supported for new and existing Fivetran integrations.
• Metadata API — only supported for existing Fivetran integrations, you will not be able to create a new connection using the Metadata API extraction method in Atlan. Atlan recommends transitioning to the Fivetran Platform Connector in Fivetran and Atlan:
  • Fivetran has already deprecated its Metadata API and released a more comprehensive Platform Connector.
  • If Fivetran revokes API access at a future date, any existing Fivetran workflows in Atlan configured with the Metadata API will fail and Atlan will not receive any new Fivetran lineage metadata for your assets.

Migrating to Fivetran Platform Connector

To set up or migrate to the Fivetran Platform Connector:

1. Configure the Platform Connector in Fivetran.
   1. Create a destination in Fivetran to receive log events. This destination must be supported in Atlan for Fivetran metadata enrichment. Skip to the next step if already configured. 
   2. Set up your Fivetran Platform Connector account-wide.
2. Once the Fivetran Platform Connector has been set up and run successfully in Fivetran, you will need to create a connection in Atlan for the destination warehouse you configured.
3. Next, you can either:
   
   • If you do not have an existing Fivetran workflow, create a new Fivetran workflow in Atlan to crawl metadata from the Fivetran Platform Connector.
   • If you have an existing Fivetran workflow, modify the Fivetran crawler configuration in Atlan to update the extraction method from Metadata API to Fivetran Platform Connector. Refer to How to crawl Fivetran for the inputs required.

Benefits of migration

You can expect the following after you have migrated from the Metadata API to Fivetran Platform Connector in Fivetran and Atlan, respectively:

• Atlan will use Fivetran Platform Connector logs to generate lineage, replacing the previously used Metadata API.
• There will be no impact on existing Fivetran lineage in Atlan if the corresponding connector in Fivetran is synced regularly. Atlan will only incrementally enrich your existing assets with updates from Fivetran.
• Any stale existing Fivetran lineage for which the corresponding connector in Fivetran is not synced regularly will be removed in Atlan. Learn more about data retention period and type of sync in Fivetran documentation.
• New connector assets will be created, with usage metrics and metadata extracted from the Fivetran Platform Connector.
• For any Fivetran sources or destinations that are not natively supported or are supported but have not been crawled yet, Atlan will create partial assets to provide you with a complete view of lineage.

Fivetran Platform Connector

Create a destination

The Fivetran Platform Connector delivers your logs and account or destination metadata to a schema in your destination. Fivetran automatically adds this connector to every new destination you create. You will need to set up at least one destination to receive the log events.

Atlan currently supports the following destinations, refer to Fivetran documentation linked below to configure them in Fivetran:

• Amazon Redshift
• Databricks
• Google BigQuery
• PostgreSQL
• Snowflake

If you have already configured a destination in Fivetran, skip to the next step.

Configure Fivetran Platform Connector

Once you have set up a supported destination, follow the steps in this setup guide from Fivetran to set up your Fivetran Platform Connector account-wide.

• You must configure the Fivetran Platform Connector on an account level.
• Atlan recommends not excluding any columns to ensure that the Fivetran enrichment package runs smoothly.

• The warehouse credentials or role configured in Atlan must have the necessary permissions to query tables in <Fivetran_destination_database>.<FPC_schema_name>. Refer to the Fivetran documentation for available FPC_schema_name values.

  The role must have the required permissions to access these tables. Refer to the relevant guide below for setting up permissions:

  • Set up Snowflake
  • Set up Google BigQuery
  • Set up Databricks
  • Set up Amazon Redshift
  • Set up PostgreSQL

This will enable you to sync all the metadata and logs for all the connectors in your account to a single destination.