1. Atlan
  2. Connections
  3. Google BigQuery Connectivity

What does Atlan crawl from Google BigQuery?

Updated

Once you have crawled Google BigQuery, you can use connector-specific filters for quick asset discovery. The following filters are currently supported for these assets:

  • Tables — BigQuery labels filter

Atlan does not run any table scans. Atlan leverages the table preview options from Google BigQuery that allow you to view data for free and without affecting any quotas using the tabledata.list API. Hence, table asset previews in Atlan are already cost-optimized. However, this does not apply to views and materialized views.

For Google BigQuery views and materialized views, Atlan will send you a cost nudge before viewing a sample data preview. This will inform you about the precise bytes that will be spent during the execution of the query, helping you decide if you would still like to run the preview.

💪 Did you know? You will also receive a cost nudge before querying your Google BigQuery assets.

 

Atlan crawls and maps the following assets and properties from Google BigQuery.

Databases

Atlan maps projects from Google BigQuery to its Database asset type.

Source property Atlan property Where in Atlan
Project ID name asset preview, profile, and filter, overview sidebar

Schemas

Atlan maps datasets from Google BigQuery to its Schema asset type.

Source property Atlan property Where in Atlan
TABLE_SCHEMA name asset preview and profile, overview sidebar
TABLE_COUNT tableCount asset preview and profile
VIEW_COUNT viewsCount asset preview and profile, overview sidebar
TABLE_CATALOG databaseName asset preview
REMARKS description asset preview and profile, overview sidebar
CREATED sourceCreatedAt asset profile and properties sidebar
MODIFIED sourceUpdatedAt asset profile and properties sidebar

Tables

💪 Did you know? Table asset previews are already cost-optimized. Google BigQuery allows you to use the table preview options to view data for free and without affecting any quotas. Note that this is currently not supported for Google BigQuery views and materialized views in Atlan.

Atlan maps tables from Google BigQuery to its Table asset type.

Source property Atlan property Where in Atlan
TABLE_NAME name asset preview and profile, overview sidebar
REMARKS description asset preview and profile, overview sidebar
COLUMN_COUNT columnCount asset preview and profile, overview sidebar
ROW_COUNT rowCount asset preview, profile, and filter, overview sidebar
SIZE_BYTES sizeBytes asset filter and overview sidebar
TABLE_TYPE subType asset preview and profile
LABELS assetTags overview sidebar
CREATED sourceCreatedAt asset profile and properties sidebar
MODIFIED sourceUpdatedAt asset profile and properties sidebar
OPTION_NAMES (require_partition_filter) isPartitioned API only

Views

Atlan maps views from Google BigQuery to its View asset type.

Source property Atlan property Where in Atlan
TABLE_NAME name asset preview and profile, overview sidebar
REMARKS description asset preview and profile, overview sidebar
COLUMN_COUNT columnCount asset preview and profile, overview sidebar
TABLE_TYPE subType asset preview and profile
CREATED sourceCreatedAt asset profile and properties sidebar
MODIFIED sourceUpdatedAt asset profile and properties sidebar
OPTION_NAMES (require_partition_filter) isPartitioned API only
DDL definition asset profile and overview sidebar

Materialized views

Atlan maps materialized views from Google BigQuery to its MaterialisedView asset type.

Source property Atlan property Where in Atlan
TABLE_NAME name asset preview and profile, overview sidebar
REMARKS description asset preview and profile, overview sidebar
COLUMN_COUNT columnCount asset preview and profile, overview sidebar
ROW_COUNT rowCount asset preview, profile, and filter, overview sidebar
SIZE_BYTES sizeBytes asset filter and overview sidebar
TABLE_TYPE subType asset preview and profile
CREATED sourceCreatedAt asset profile and properties sidebar
MODIFIED sourceUpdatedAt asset profile and properties sidebar
OPTION_NAMES (require_partition_filter) isPartitioned API only
DDL definition asset profile and overview sidebar

Columns

💪 Did you know? Atlan supports nested columns up to level 1 for Google BigQuery to help you enrich your semi-structured data types. You can view nested columns in the asset sidebar for your table assets.

Atlan maps columns from Google BigQuery to its Column asset type.

Source property Atlan property Where in Atlan
COLUMN_NAME name asset preview and profile, overview sidebar
REMARKS, DESCRIPTION description asset preview and profile, overview sidebar
ORDINAL_POSITION order asset profile
TYPE_NAME dataType asset preview, profile, and filter, overview sidebar
IS_NULLABLE isNullable API only
IS_PARTITIONING_COLUMN isPartition asset preview, profile, and filter
CLUSTERING_COLUMN_LIST isClustered asset preview, profile, and filter

Stored procedures

Atlan maps stored procedures from Google BigQuery to its Procedure asset type.

Source property Atlan property Where in Atlan
PROCEDURE_NAME name API only
REMARKS description API only
PROCEDURE_TYPE subType API only
ROUTINE_DEFINITION definition API only
CREATED sourceCreatedAt API only
MODIFIED sourceUpdatedAt API only

Related articles

Was this article helpful?
1 out of 1 found this helpful