How to connect on-premises databases to Kubernetes

You can configure and use Atlan's metadata-extractor tool to extract metadata from on-premises databases with Kubernetes deployment architecture, as an alternative to using Docker Compose.

Get the metadata-extractor tool

To get the metadata-extractor tool:

1. Raise a support ticket to get a link to the latest version.
2. Download the image using the link provided by support.

Load image to Kubernetes cluster

You cannot upload the extractor image directly to a Kubernetes cluster. You must upload the extractor image to a container registry that your Kubernetes cluster can access. This ensures that Kubernetes can readily deploy pods with the metadata-extractor tool.

Apply configuration maps

ConfigMaps contain essential settings that enable the metadata-extractor tool to connect to your database, including connection details and extraction parameters. This can help you customize the extraction process to fit your database environment.

1. Deploy configurations to your specific database setup:

   kubectl apply -f config-maps.yml

2. Create a config-maps.yml containing your database settings, for example:

   apiVersion: v1
   kind: ConfigMap
   metadata:
     name: atlan-extractor-config-mysql
   data:
     DOWNLOAD_JDBC: "true"
     DOWNLOAD_JDBC_URL: "https://example.com/path/to/jdbc-driver.tar.gz"
     DRIVER: "com.example.jdbc.Driver"
     # Add other necessary configurations as key-value pairs

   • Replace example values with details of your database connection and the JDBC driver.

Deploy extraction job

Set up the CronJob for metadata extraction from the database:

kubectl apply -f job.yml

Example

Create a job.yml for the extraction job with details like the following:

apiVersion: batch/v1
kind: CronJob
metadata:
  name: atlan-extractor-cron-job
spec:
  schedule: "@weekly"
  jobTemplate:
    spec:
      template:
        spec:
          containers:
            - name: crawler
              image: your-registry/path-to-extractor-image:latest
              # Define environment variables and volume mounts as required

(Optional) Configure CronJob schedule

The CronJob is configured to execute weekly by default.

To configure the CronJob schedule:

1. Open the job.yml file.
2. In the spec section, for schedule:, replace the "@weekly" cron expression with your preferred schedule. For example, use "@daily" for daily executions or provide a custom cron schedule.

For more information on CronJob schedules, refer to Kubernetes documentation.

(Optional) Trigger the job manually

To trigger an immediate metadata extraction, execute the CronJob manually:

kubectl create job --from=cronjob/atlan-extractor-cron-job crawl-mysql-$(date '+%Y-%m-%d-%H-%M-%S')

Request files from Atlan

To get started, contact Atlan support to request sample ConfigMap and CronJob files for supported SQL connectors:

• Microsoft SQL Server
• MySQL
• Oracle
• PostgreSQL