The Atlan Secure Agent is a Kubernetes-based application that runs within a customer's environment. It acts as a gateway between the single-tenant Atlan SaaS and external systems like Snowflake, Tableau, and other data sources. This document explains the Secure Agent's deployment architecture, key components, communication flows, and security considerations.
High-level architecture
This section describes how the Secure Agent is structured and deployed. It explains the core components that enable metadata extraction, job execution, and communication with Atlan.
Core components
The Secure Agent runs as a Kubernetes-based application within a customer's private cloud or on-premises environment. It consists of several key components that work together to execute metadata extraction tasks.
Argo Workflows
- An Argo Workflow server is deployed to coordinate all activities and launch Kubernetes workloads.
- The Secure Agent uses Argo Workflows to orchestrate and manage metadata extraction jobs.
- Each workflow represents a unit of work, such as extracting metadata from a source system.
Agent orchestrator
- A scheduled job that runs every five minutes to check for jobs that need to be executed.
- It connects to the Atlan tenant, retrieves job details, and initiates workflows accordingly.
Auxiliary services
Additional services that support agent operations:
- Health monitoring service sends periodic heartbeats to Atlan to confirm the agent is active.
- Logging service uploads execution logs to Atlan for monitoring and debugging.
Metadata extraction workflows
- Connector-specific jobs that extract metadata from source systems.
- Workflows run in isolated containers, ensuring security and reliability.
Data flow
The Secure Agent supports two modes of metadata transfer. Each mode determines how extracted metadata is delivered to Atlan.
Bucket relay
Bucket relay is the recommended mode for metadata extraction. In this mode, metadata is first stored in enterprise-managed cloud storage before Atlan retrieves it.
- The Secure Agent extracts metadata and writes it to a storage bucket in the customer’s cloud environment (such as AWS S3, Azure Blob Storage, or Google Cloud Storage). This is managed by providing the agent write access to cloud storage.
- Atlan retrieves metadata from the storage bucket and processes it further. This is managed by providing Atlan read access to list and read files in cloud storage.
- This mode ensures the extracted data remains within the customer’s infrastructure until Atlan explicitly fetches it. Customers can also use this data for auditing.
Direct ingestion
In direct ingestion mode, metadata is transferred directly from the Secure Agent to Atlan.
- The Secure Agent uses pre-signed URLs to upload metadata directly to Atlan. Some cloud storage providers that use pre-signed URLs include:
- A pre-signed URL grants temporary access to upload files without exposing long-term credentials.
- Each URL has an expiration time, ensuring access is only available for a limited duration.
See also
- Secure Agent - Security: Details on security mechanisms.