What does Atlan crawl from Google Cloud Composer/OpenLineage?

Once you have integrated Google Cloud Composer/OpenLineage, you can use connector-specific filters for quick asset discovery. The following filters are currently supported:

  • Status filter — last run status for an asset
  • Duration filter — last run duration for an asset

Atlan maps the following assets and properties from Google Cloud Composer/OpenLineage. Asset lineage support depends on the list of operators supported by OpenLineage.

DAGs

Atlan maps DAGs (directed acyclic graphs) from Google Cloud Composer/OpenLineage to its AirflowDAG asset type.

Source property Atlan property
schedule_interval airflowDagSchedule
timetable airflowDagScheduleDelta
tags airflowTags
version airflowRunVersion
openlineageAdapterVersion airflowRunOpenLineageVersion
runid airflowRunName
run_type airflowRunType
eventTime airflowRunStartTime
eventTime airflowRunEndTime
eventType airflowRunOpenLineageState
query airflowTaskSql
group_id airflowTaskGroupName

Tasks

Atlan maps tasks from Google Cloud Composer/OpenLineage to its AirflowTask asset type.

Source property Atlan property
retries airflowTaskRetryNumber
pool airflowTaskPool
pool_slots airflowTaskPoolSlots
queue airflowTaskQueue
priority_weight airflowTaskPriorityWeight
trigger_rule airflowTaskTriggerRule
operator_class airflowTaskOperatorClass
dag_id airflowDagName
conn_id airflowTaskConnectionId
sql airflowTaskSql
tags airflowTags
version airflowRunVersion
openlineageAdapterVersion airflowRunOpenLineageVersion
runid airflowRunName
run_type airflowRunType
eventTime airflowRunStartTime
eventTime airflowRunEndTime
eventType airflowRunOpenLineageState

Related articles

Was this article helpful?
0 out of 0 found this helpful