How to mine Snowflake

Once you have crawled assets from Snowflake, you can mine its query history to construct lineage.

To mine lineage from Snowflake, review the order of operations and then complete the following steps.

Select the miner

To select the Snowflake miner:

1. In the top right of any screen, navigate to New and then click New Workflow.
2. From the filters along the top, click Miner.
3. From the list of packages, select Snowflake Miner and then click Setup Workflow.

Configure the miner

To configure the Snowflake miner:

1. For Connection, select the connection to mine. (To select a connection, the crawler must have already run.)
2. For Miner Extraction Method, select Source, or see the separate instructions for the S3 miner.
3. For Snowflake Database:
   • If the connection is configured with access to the snowflake database, choose Default.
   • If the connection can only access a separate cloned database, choose Cloned Database.
4. If you are using a cloned database, enter the name of the cloned database in Database Name and the name of the cloned schema in Schema Name.
5. For Start time, choose the earliest date from which to mine query history.
   💪 Did you know? The miner restricts you to only querying the past two weeks of query history. If you need to query more history, for example in an initial load, consider using the S3 miner first. After the initial load, you can modify the miner's configuration to use query history extraction.
6. To check for any permissions or other configuration issues before running the miner, click Preflight checks.
7. At the bottom of the screen, click Next to proceed.

Configure the miner behavior

To configure the Snowflake miner behavior:

1. (Optional) For Calculate popularity, keep True to retrieve usage and popularity metrics for your Snowflake assets from query history.
   • For Excluded Users, type the names of users to be excluded while calculating usage metrics for Snowflake assets. Press Enter after each name to add more names. 
2. (Optional) For Advanced Config, keep Default for the default configuration or click Custom to configure the miner:
   • If Atlan support has provided you with a custom control configuration, enter the configuration into the Custom Config box.
   • You can also enter {“ignore-all-case”: true} to enable crawling assets with case-sensitive identifiers.
   • For Popularity Window (days), 90 days is the maximum limit. You can set a shorter popularity window of less than 90 days.

Run the miner

To run the Snowflake miner, after completing the steps above:

• To run the miner once immediately, at the bottom of the screen, click the Run button.
• To schedule the miner to run hourly, daily, weekly, or monthly, at the bottom of the screen, click the Schedule & Run button.

Once the miner has completed running, you will see lineage for Snowflake assets that were created in Snowflake between the start time and when the miner ran! 🎉