How can Atlan generate lineage upstream from the data warehouse layer?

 

Replication Type Constraint Recommended Path
Custom Jobs These jobs don’t have an interface for Atlan to extract lineage information from (i.e. custom python jobs that move data from RDBMS to S3 to Snowflake).  Use Atlan's Lineage API to push source-to-target mappings into Atlan.
Data Integration tools These tools historically have not had Metadata APIs for Atlan to extract lineage information from. Atlan is releasing support for data integration tools that have released Metadata APIs (such as Fivetran).
Orchestration tools These tools (i.e. Airflow) are not asset-aware so there is no way for Atlan to extract lineage information as the system does not know the source and target objects that a DAG is operating on. Use Atlan's Lineage API to push source-to-target mappings into Atlan.

 

More about Atlan's Lineage API

Atlan has 2 API Endpoints — one for tables/views/objects and one for columns/fields. The input to the API Endpoint is the source (i.e. Table A and Table B) to target (i.e Table C) mapping and outputs an object (ProcessEntity) that forms the lineage relationship between those objects.

Related articles

Was this article helpful?
0 out of 0 found this helpful