What does Atlan crawl from Apache Spark/OpenLineage?

🧪 Preview feature! This feature is available for your experimentation, and we'd love your feedback. It may change before its final generally-available form. If you'd like to participate in the preview, reach out to your customer success manager for more information.

Once you have integrated Apache Spark/OpenLineage, you can use connector-specific filters for quick asset discovery. The following filters are currently supported:

  • Status filter — last run status for an asset
  • Duration filter — last run duration for an asset

Atlan maps the following assets and properties from Apache Spark/OpenLineage.

Jobs

Atlan maps jobs from Apache Spark to its SparkJob asset type.

Source property Atlan property
appName sparkAppName
master sparkMaster

OpenLineage metadata

Atlan reports OpenLineage operational metadata for Spark jobs.

Atlan property Description
sparkRunVersion Spark runtime version
sparkRunOpenLineageVersion OpenLineage library version
sparkRunStartTime job start time
sparkRunEndTime job end time
sparkRunOpenLineageState status of the job

Related articles

Was this article helpful?
0 out of 0 found this helpful