Replication Type | Constraint | Recommended Path |
Custom Jobs | These jobs don’t have an interface for Atlan to extract lineage information from (i.e. custom python jobs that move data from RDBMS to S3 to Snowflake). | Use Atlan's Lineage API to push source-to-target mappings into Atlan. |
Data Integration tools | These tools historically have not had Metadata APIs for Atlan to extract lineage information from. | Atlan is releasing support for data integration tools that have released Metadata APIs (such as Fivetran). |
Orchestration tools | These tools (i.e. Airflow) are not asset-aware so there is no way for Atlan to extract lineage information as the system does not know the source and target objects that a DAG is operating on. | Use Atlan's Lineage API to push source-to-target mappings into Atlan. |
More about Atlan's Lineage API
Atlan has 2 API Endpoints — one for tables/views/objects and one for columns/fields. The input to the API Endpoint is the source (i.e. Table A and Table B) to target (i.e Table C) mapping and outputs an object (ProcessEntity) that forms the lineage relationship between those objects.