Atlan architecture

Have more questions? Submit a request

architecture.png

The components of Atlan are isolated, across both compute and data. For more detail, see How are resources isolated?

Platform components

  • Kong is an API gateway. It handles rate limiting and token verification on all incoming API requests.
  • Apache Keycloak is an identity and access management component. It manages everything to do with users, login, SSO and so on.
  • Heracles is Atlan's API service. It houses the business logic used by the frontend and APIs to interact with other platform components.
  • PostgreSQL is a SQL database. Many services on the platform use it for storage.
  • HashiCorp Vault is a secret manager. It stores sensitive credentials provided by the user.
  • Apache Ranger is the policy engine. It provides fine-grained access control over data in the metastore.
  • Argo Workflows is a workflow orchestrator for k8s. It runs and manages long-running jobs in a container and k8s-native fashion.
  • Admission Controller is a k8s admission controller. It performs certain actions when Argo Workflows are updated such as workflow alerts.
  • Metastore stores metadata as data in a graph store. It is based on Apache Atlas and has fine-grained access control on top.
    • Apache Zookeeper manages consensus and coordination for the metastore services.
    • Elasticsearch indexes data and drives search functionality.
    • Apache Cassandra is an object-oriented database used to store the metastore's data.
  • Apache Kafka is an event stream. It enables event-driven use cases across the platform.
  • Heka is Atlan's SQL component. It parses, rewrites and optimizes SQL queries and is powered by Apache Calcite.
  • Redis is a cache layer used by Heracles.

Platform management components

  • Velero performs cluster backups.
  • Rancher is a cluster manager. It performs cluster actions after receiving instructions from the central Rancher Instance Manager.
  • Alertmanager sends alerts generated by metrics stored in Prometheus.
  • Grafana provides observability dashboards.
  • Prometheus is a time-series database to store metrics.
  • Kibana explores and filters log data stored in Elasticsearch.
  • Fulentbit is a logging and metrics processor. It parses and pushes logs from pods to various destinations.
  • Elasticsearch stores and indexes logs.

Central components

  • Pager Duty is used for incident response. We get alerts when something goes wrong in one of the clusters.
  • Argo CD is used for continuous deployment. Changes in our git repositories lead to upgrades in the clusters.
  • Github Actions update the Docker container images as part of our development processes.
  • Sendgrid is used to send emails.
  • The frontend is a Vue.js web application that is hosted on S3 and delivered via Amazon CloudFront content delivery network (CDN) service.

Atlan marketplace (not pictured)

Our marketplace of packages (workflows) that perform long-running tasks on the Atlan platform. We use this ecosystem of packages to build our metadata and lineage connectors.

Related articles

Was this article helpful?
1 out of 1 found this helpful