What does Atlan crawl from Microsoft Azure Data Factory?

Atlan crawls and maps the following assets and properties from Microsoft Azure Data Factory.

For any currently unsupported linked services, datasets, and activities from Microsoft Azure Data Factory not listed below, Atlan will map them to the relevant asset type and only display asset name, type, and description.

Linked services

Atlan maps linked services from Microsoft Azure Data Factory to its AdfLinkedservice asset type.

Microsoft Azure Cosmos DB for MongoDB

Source property Atlan property
name name
type adfLinkedserviceType
description description
annotations adfLinkedserviceAnnotations
typeProperties.database adfLinkedserviceDatabaseName
typeProperties.isServerVersionAbove32 adfLinkedserviceVersionAbove
version adfLinkedserviceVersion

Azure Data Lake Storage (ADLS)

Source property Atlan property
name name
type adfLinkedserviceType
description description
typeProperties.azureCloudType adfLinkedserviceAzureCloudType
typeProperties.servicePrincipalCredentialType adfLinkedserviceCredentialType
typeProperties.tenant adfLinkedserviceTenant
typeProperties.url adfLinkedserviceDomainEndpoint

Azure Databricks

Source property Atlan property
name name
type adfLinkedserviceType
description description
annotations adfLinkedserviceAnnotations
typeProperties.clusterId adfLinkedserviceClusterId
typeProperties.domain adfLinkedserviceDomainEndpoint
typeProperties.workspaceResourceId adfLinkedserviceResourceId

Snowflake

Source property Atlan property
name name
type adfLinkedserviceType
description description
annotations adfLinkedserviceAnnotations
typeProperties.accountIdentifier adfLinkedserviceDomainEndpoint
typeProperties.authenticationType adfLinkedserviceCredentialType
typeProperties.database adfLinkedserviceDatabaseName
typeProperties.user adfLinkedserviceUserName
typeProperties.warehouse adfLinkedserviceWarehouseName

Microsoft Azure SQL Database

Source property Atlan property
name name
type adfLinkedserviceType
description description
annotations adfLinkedserviceAnnotations
typeProperties.authenticationType adfLinkedserviceCredentialType
typeProperties.database adfLinkedserviceDatabaseName
typeProperties.server adfLinkedserviceDomainEndpoint

Datasets

Atlan maps databases from Microsoft Azure Cosmos DB to its AdfDataset asset type.

Microsoft Azure Cosmos DB for MongoDB

Source property Atlan property
name name
type adfDatasetType
description description
annotations adfDatasetAnnotations
folder adfDatasetFolderPath
linkedServiceName adfDatasetLinkedService
typeProperties.collection adfDatasetCollectionName

Azure Data Lake Storage (ADLS)

Source property Atlan property
name name
type adfDatasetType
description description
annotations adfDatasetAnnotations
folder adfDatasetFolderPath
linkedServiceName adfDatasetLinkedService
typeProperties.fileName adfDatasetFileName
typeProperties.folderPath adfDatasetFileFolderPath
typeProperties.format adfDatasetStorageType
typeProperties.container adfDatasetContainerName

Azure Databricks

Source property Atlan property
name name
type adfDatasetType
description description
annotations adfDatasetAnnotations
folder adfDatasetFolderPath
linkedServiceName adfDatasetLinkedService
typeProperties.database adfDatasetDatabaseName
typeProperties.table adfDatasetTableName

Snowflake

Source property Atlan property
name name
type adfDatasetType
description description
annotations adfDatasetAnnotations
folder adfDatasetFolderPath
linkedServiceName adfDatasetLinkedService
typeProperties.schema adfDatasetSchemaName
typeProperties.table adfDatasetTableName

Microsoft Azure SQL Database

Source property Atlan property
name name
type adfDatasetType
description description
annotations adfDatasetAnnotations
folder adfDatasetFolderPath
linkedServiceName adfDatasetLinkedService
typeProperties.schema adfDatasetSchemaName
typeProperties.table adfDatasetTableName

Data flows

Atlan maps data flows from Microsoft Azure Data Factory to its AdfDataflow asset type.

Source property Atlan property
name name
description description
folder adfDataflowFolderPath
typeProperties.sources adfDataflowSources
typeProperties.sinks adfDataflowSinks
typeProperties.scriptLines adfDataflowScript

Activities

Atlan maps activities from Microsoft Azure Data Factory to its AdfActivity asset type.

Copy activity

Source property Atlan property
name name
type adfActivityType
description description
dependsOn adfActivityPrecedingDependency
policyTimeout adfActivityPolicyTimeout
policyRetryInterval adfActivityPolictRetryInterval
state adfActivityState
activityRuns adfActivityRuns
typeProperties.sink adfActivitySinks
typeProperties.source adfActivitySources
typeProperties.sinkType adfActivitySinkType
typeProperties.sourceType adfActivitySourceType

Databricks notebooks

Source property Atlan property
name name
type adfActivityType
description description
dependsOn adfActivityPrecedingDependency
policyTimeout adfActivityPolicyTimeout
policyRetryInterval adfActivityPolictRetryInterval
state adfActivityState
activityRuns adfActivityRuns
typeProperties.source adfActivitySources
typeProperties.sourceType adfActivitySources
typeProperties.notebookPath adfActivityNotebookPath

DatabricksSparkJar activity

Source property Atlan property
name name
type adfActivityType
description description
dependsOn adfActivityPrecedingDependency
policyTimeout adfActivityPolicyTimeout
policyRetryInterval adfActivityPolictRetryInterval
state adfActivityState
activityRuns adfActivityRuns
typeProperties.source adfActivitySources
typeProperties.sourceType adfActivitySources
typeProperties.mainClassName adfActivityMainClassName

DatabricksSparkPython activity

Source property Atlan property
name name
type adfActivityType
description description
dependsOn adfActivityPrecedingDependency
policyTimeout adfActivityPolicyTimeout
policyRetryInterval adfActivityPolictRetryInterval
state adfActivityState
activityRuns adfActivityRuns
typeProperties.source adfActivitySources
typeProperties.sourceType adfActivitySources
typeProperties.pythonFile adfActivityPythonFilePath

Lookup activity

Source property Atlan property
name name
type adfActivityType
description description
dependsOn adfActivityPrecedingDependency
policyTimeout adfActivityPolicyTimeout
policyRetryInterval adfActivityPolictRetryInterval
state adfActivityState
activityRuns adfActivityRuns
typeProperties.firstRowOnly adfActivityFirstRowOnly
typeProperties.source adfActivitySources

ForEach activity

Source property Atlan property
name name
type adfActivityType
description description
dependsOn adfActivityPrecedingDependency
policyTimeout adfActivityPolicyTimeout
policyRetryInterval adfActivityPolictRetryInterval
state adfActivityState
activityRuns adfActivityRuns
typeProperties.activities adfActivitySubActivities
typeProperties.batchCount adfActivityBatchCount
typeProperties.isSequential adfActivityIsSequential

Pipelines

Atlan maps pipelines from Microsoft Azure Data Factory to its AdfPipeline asset type.

Source property Atlan property
name name
type adfActivityType
description description
activityCount adfPipelineActivityCount
pipelineRun adfPipelineRuns
typeProperties.folder adfPipelineFolderPath

Related articles

Was this article helpful?
0 out of 0 found this helpful