Atlan can help you understand the events that occur in your tenants, including user and administrative actions. Learn more about logging and retention as follows:
Tenant logs
Note the following:
- Load balancer logs for Azure and GCP tenants are currently not enabled.
- AuditSearch and SearchLog records are persisted forever in Elasticsearch. The 30-day retention period pertains to application logs written to logging Elasticsearch.
- An example of block storage mentioned below is Amazon Elastic Block Store (EBS) for AWS.
Production tenants
| Log types | Retention | Storage | AWS | Azure | GCP |
|---|---|---|---|---|---|
| Active tenant overall backup | 15 days | Object storage | ✅ | ✅ | ✅ |
| Offboarded tenant overall backup | AWS — 30 days, Azure and GCP — 15 days | Object storage | ✅ | ✅ | ✅ |
| Load balancer logs | 30 days | Object storage | ✅ | ❌ | ❌ |
| Audit — user events | 60 days | PostgreSQL | ✅ | ✅ | ✅ |
| Audit — admin events | Unlimited | PostgreSQL | ✅ | ✅ | ✅ |
| Application logs | 30 days | Elasticsearch and object storage | ✅ | ✅ | ✅ |
| Workflow logs | 90 days | ClickHouse | ✅ | ✅ | ✅ |
| Workflow artifacts | 180 days | Object storage | ✅ | ✅ | ✅ |
| Application metrics | 60 days | VictoriaMetrics (block storage) | ✅ | ✅ | ✅ |
Proof of value (POV) tenants
| Log types | Retention | Storage | AWS | Azure | GCP |
|---|---|---|---|---|---|
| Active tenant overall backup | 15 days | Object storage | ✅ | ✅ | ✅ |
| Offboarded tenant overall backup | AWS — 3 days, Azure and GCP — 15 days | Object storage | ✅ | ✅ | ✅ |
| Load balancer logs | 30 days | Object storage | ✅ | ❌ | ❌ |
| Audit — user events | 60 days | PostgreSQL | ✅ | ✅ | ✅ |
| Audit — admin events | Unlimited | PostgreSQL | ✅ | ✅ | ✅ |
| Application logs | 30 days | Elasticsearch and object storage | ✅ | ✅ | ✅ |
| Workflow artifacts | 180 days | Object storage | ✅ | ✅ | ✅ |
| Application metrics | 60 days | VictoriaMetrics (block storage) | ✅ | ✅ | ✅ |
Atlan logs
| Service | Type | Logging pipeline | Destination |
|---|---|---|---|
| Heracles | application | Fluent Bit | S3 |
| Argo | application, server | Argo, Fluent Bit | S3 |
| Atlas | application, audit, perf | Fluent Bit | S3 |
| Numaflow | application | Fluent Bit | S3 |
| Kube events | application | Fluent Bit | S3 |
| Wisdom | application, audit | Fluent Bit | S3 |
| Chronos | application | Fluent Bit | S3 |
| Redis | application | Fluent Bit | S3 |
| Kong | application, audit | Fluent Bit, PostgreSQL, Keycloak REST API | S3 |
| Keycloak | application | Fluent Bit | S3 |
| Elasticsearch | application | Fluent Bit | S3 |
| Cassandra | application | Fluent Bit | S3 |
| Heka | application | Fluent Bit | S3 |
| Pgpool | application, server | Fluent Bit | S3 |
| Kafka | events | Fluent Bit | S3 |
Cloud storage lifecycle
The cloud storage created for each tenant has its own lifecycle. The lifecycle policy is attached to paths in the cloud storage. The lifecycle policy applied to a production tenant is as follows:
Amazon Web Services (AWS)
| Lifecycle policy | Path | Action |
|---|---|---|
DeleteClusterLogsAfter30Days |
logs/ |
Expires |
DeleteArgoArtifactsAfter180Days |
argo-artifacts/ |
Transition to S3 Glacier Flexible Retrieval, then expires |
DeleteArgoBackupAfter15Days |
backup/argo/ |
Expires |
DeleteAltanScheduleQuery |
argo-artifacts/default/schedule-query/ |
Expires |
DeletePostgresBackupAfter15Days |
backup/postgres/ |
Expires |
DeleteRedisBackupAfter15Days |
backup/redis/ |
Expires |
DeleteCassandraBackupAfter15Days |
backup/cassandra/ |
Expires |
DeletePrometheusBackupAfter15Days |
backup/prometheus/ |
Expires |
DeleteALBLogsAfter30Days |
AWSLogs/ |
Expires |
Microsoft Azure
| Lifecycle policy | Path | Action |
|---|---|---|
DeleteClusterLogsAfter30Days |
logs/ |
Delete |
DeleteArgoArtifactsAfter180Days |
argo-artifacts/ |
Moves to archive after 90 days and delete after 180 days |
DeleteArgoBackupAfter15Days |
backup/argo/ |
Delete |
DeletePostgresBackupAfter15Days |
backup/postgres/ |
Delete |
DeleteRedisBackupAfter15Days |
backup/redis/ |
Delete |
DeleteCassandraBackupAfter15Days |
backup/cassandra/ |
Delete |
DeleteAltanScheduleQuery |
argo-artifacts/default/schedule-query/ |
Delete if blobs not modified in 1 day |
DeleteSparkEventLogsAfter15Days |
spark-event-logs/ |
Delete |
Google Cloud Platform (GCP)
| Lifecycle policy | Path | Action |
|---|---|---|
|
|
logs/ |
Delete |
180+ days since object was created
Name matches prefix 'argo-artifacts/' |
argo-artifacts/ |
Archive |
270+ days since object was created
Name matches prefix 'argo-artifacts/' |
argo-artifacts/ |
Delete |
3+ days since object was created
Name matches prefix 'backup/argo/' |
backup/argo/ |
Delete |
3+ days since object was created
Name matches prefix 'backup/postgres/' |
backup/postgres/ |
Delete |
3+ days since object was created
Name matches prefix 'backup/prometheus/' |
backup/prometheus/ |
Delete |
3+ days since object was created
Name matches prefix 'backup/redis/' |
backup/redis/ |
Delete |
1+ days since object was created
Name matches prefix 'argo-artifacts/default/schedule-query/' |
argo-artifacts/default/schedule-query/ |
Delete |
15+ days since object was created
Name matches prefix 'spark-event-logs/' |
spark-event-logs/ |
Delete |
3+ days since object was created
Name matches prefix 'backup/cassandra/' |
backup/cassandra/ |
Delete |