Does Atlan support Spark SQL?
Yes, Atlan supports Spark SQL through Spark jobs.
How does Atlan handle multiple DAG owner email addresses?
Atlan captures up to the first 10 valid owner email addresses for a DAG. If a DAG has more than 10 owner emails, only the first 10 are stored as “Source Owners” in Atlan. To capture DAG owners, make sure to set the owner field when defining the DAG in Apache Airflow. Ensure that the owner emails are comma-separated and valid to be recognized correctly.
Can I connect multiple Apache Airflow instances to a single Atlan instance?
Yes. However, you will need to create a separate connection for each Apache Airflow instance you want to connect to Atlan.
Why are some Apache Airflow assets or lineage missing even after the workflow ran successfully?
If OpenLineage has not been configured properly, it may have been unable to send any events while the DAG ran. You can view event logs in Atlan to track and debug events received from OpenLineage. However, if you have verified that your OpenLineage connection was configured correctly and events are still missing, please reach out to your customer success manager at Atlan or raise a support ticket.
Does Atlan support Column Level Lineage (CLL) for object storage?
Atlan currently does not support Column Level Lineage (CLL) for object storage. This is because object storage systems do not have structured schema, unlike relational data sources.
Object storage systems store unstructured data, unlike relational data sources where columns and relationships are clearly defined. As a result, object storage systems cannot support column level lineage. For example, unstructured data can include a collection of image files stored in an S3 bucket, which doesn't support column-level lineage.
To enable CLL for object storage, users must register S3 objects as tables using AWS Glue, Hive, or similar cataloging tools.
Column level lineage support is also not available for the following Apache Airflow distributions: