Atlan’s lineage generation is based on SQL-parsing. Parsing of Spark jobs is currently not supported. However, Databricks has released their Unity Catalog solution that enables lineage generation from Spark jobs that occur in Databricks. Atlan offers native integration with the Unity Catalog API to ingest lineage from Spark transformations in Databricks.
