Once you have crawled assets from Google BigQuery you can mine its query history to construct lineage.
To mine lineage from Google BigQuery, complete the following steps.
Select the miner
To select the Google BigQuery miner:
- In the top right of any screen, navigate to New and then click New Workflow.
- From the filters along the top, click Miner.
- From the list of packages, select BigQuery Miner and click on Setup Workflow.
Configure the miner
To configure the Google BigQuery miner:
- For Connection, select the connection to mine. (To select a connection, the crawler must have already run.)
- For Miner extraction method select Query History.
- For Start time choose the earliest date from which to mine query history.
- (Optional) By default, the miner runs in the US region. To run in another region, for Region, select Custom and then enter the region under Custom BigQuery Region.
Run the miner
To run the Google BigQuery miner, after completing the steps above:
- To run the miner once, immediately, at the bottom of the screen click the Run button.
- To schedule the miner to run hourly, daily, weekly or monthly, at the bottom of the screen click the Schedule & Run button.
Once the miner has completed running, you will see lineage for Google BigQuery assets that were created in Google BigQuery between the start time and when the miner ran! 🎉