Once you have integrated Apache Spark/OpenLineage, you can use connector-specific filters for quick asset discovery. The following filters are currently supported:
- Status filter — last run status for an asset
- Duration filter — last run duration for an asset
Atlan maps the following assets and properties from Apache Spark/OpenLineage. Asset lineage support depends on the data sources that OpenLineage supports.
Jobs
Atlan maps jobs from Apache Spark to its SparkJob
asset type. Atlan also supports column-level lineage for Spark jobs.
Source property | Atlan property |
---|---|
appName |
sparkAppName |
master |
sparkMaster |
OpenLineage metadata
Atlan reports OpenLineage operational metadata for Spark jobs.
Atlan property | Description |
---|---|
sparkRunVersion |
Spark runtime version |
sparkRunOpenLineageVersion |
OpenLineage library version |
sparkRunStartTime |
job start time |
sparkRunEndTime |
job end time |
sparkRunOpenLineageState |
status of the job |