What are the known limitations of the Microsoft Azure Data Factory connector?
Following are the known limitations of the Microsoft Azure Data Factory connector:
- Lineage is not supported for data loaded through data flow activity.
- The ability to include or exclude certain pipelines or activities is currently not supported.
- Datasets defined in Microsoft Azure Data Factory are currently not mapped to actual data assets in Atlan.
- Atlan currently does not display pipelines as DAGs on the lineage graph. Only the activities that are part of a pipeline are displayed as related assets.
- Lineage is not supported for activities that do not have a defined source and sink attribute in their configuration.
Why is lineage missing?
Remember that the Microsoft Azure Data Factory package only enriches data assets that already exist in Atlan.
Check that the data assets corresponding to Microsoft Azure Data Factory's sources and sinks already exist in Atlan. For example, that the Microsoft Azure Cosmos DB assets have been crawled from Microsoft Azure Cosmos DB, Snowflake assets have been crawled from Snowflake, and so on.
The fact that a crawler has run (such as Microsoft Azure Cosmos DB or Snowflake) does not mean all assets from that source have been crawled. The filtering configuration of the crawler could have omitted assets. Or the credentials configured in the crawler may not have access to all of the assets.