What does Atlan crawl from Apache Airflow/OpenLineage?

Updated September 16, 2024 18:44

Once you have integrated Apache Airflow/OpenLineage, you can use connector-specific filters for quick asset discovery. The following filters are currently supported:

Atlan maps the following assets and properties from Apache Airflow/OpenLineage. Asset lineage support depends on the list of operators supported by OpenLineage.

DAGs

Atlan maps DAGs (directed acyclic graphs) from Apache Airflow/OpenLineage to its AirflowDAG asset type.

Source property	Atlan property
`schedule_interval`	`airflowDagSchedule`
`timetable`	`airflowDagScheduleDelta`
`tags`	`airflowTags`
`version`	`airflowRunVersion`
`openlineageAdapterVersion`	`airflowRunOpenLineageVersion`
`runid`	`airflowRunName`
`run_type`	`airflowRunType`
`eventTime`	`airflowRunStartTime`
`eventTime`	`airflowRunEndTime`
`eventType`	`airflowRunOpenLineageState`
`query`	`airflowTaskSql`
`group_id`	`airflowTaskGroupName`

Atlan maps tasks from Apache Airflow/OpenLineage to its AirflowTask asset type.

Source property	Atlan property
`retries`	`airflowTaskRetryNumber`
`pool`	`airflowTaskPool`
`pool_slots`	`airflowTaskPoolSlots`
`queue`	`airflowTaskQueue`
`priority_weight`	`airflowTaskPriorityWeight`
`trigger_rule`	`airflowTaskTriggerRule`
`operator_class`	`airflowTaskOperatorClass`
`dag_id`	`airflowDagName`
`conn_id`	`airflowTaskConnectionId`
`sql`	`airflowTaskSql`
`tags`	`airflowTags`
`version`	`airflowRunVersion`
`openlineageAdapterVersion`	`airflowRunOpenLineageVersion`
`runid`	`airflowRunName`
`run_type`	`airflowRunType`
`eventTime`	`airflowRunStartTime`
`eventTime`	`airflowRunEndTime`
`eventType`	`airflowRunOpenLineageState`