How does Atlan handle lineage from Spark jobs?

Have more questions? Submit a request

Atlan’s lineage generation is based on SQL-parsing. Parsing of Spark jobs is currently not supported. However, Databricks has released their Unity Catalog solution that enables lineage generation from Spark jobs that occur in Databricks. Atlan offers native integration with the Unity Catalog API to ingest lineage from Spark transformations in Databricks.

Related articles

Was this article helpful?
0 out of 0 found this helpful