What does Atlan crawl from Apache Spark/OpenLineage?

Once you have integrated Apache Spark/OpenLineage, you can use connector-specific filters for quick asset discovery. The following filters are currently supported:

  • Status filter — last run status for an asset
  • Duration filter — last run duration for an asset

Atlan maps the following assets and properties from Apache Spark/OpenLineage. Asset lineage support depends on the data sources that OpenLineage supports.

Jobs

Atlan maps jobs from Apache Spark to its SparkJob asset type. Atlan also supports column-level lineage for Spark jobs.

Source property Atlan property
appName sparkAppName
master sparkMaster

OpenLineage metadata

Atlan reports OpenLineage operational metadata for Spark jobs.

Atlan property Description
sparkRunVersion Spark runtime version
sparkRunOpenLineageVersion OpenLineage library version
sparkRunStartTime job start time
sparkRunEndTime job end time
sparkRunOpenLineageState status of the job

Related articles

Was this article helpful?
0 out of 0 found this helpful