Supported connections for on-premises databases

The metadata-extractor tool supports the following connection types.

These describe the details required when setting up on-premises database access.

Hive with basic authentication

Use <<: *hive under the environment section to use this connection type.

Available attributes:

• HOST - database host name or IP address (required)
• PORT - database port (optional, default is 10000)
• DEFAULT_SCHEMA - default schema name (optional, default is default)
• USERNAME - database user name (required)
• PASSWORD - database user password (required)
• SCHEMA_EXCLUDE_REGEX - regex to exclude schemas (optional)
• SCHEMA_INCLUDE_REGEX - regex to include schemas (optional)
• TEMP_TABLE_REGEX - regex to exclude tables (optional)

Example

hive-example:
    <<: *hive-extract
    environment:
      <<: *hive
      HOST: hive-host
      PORT: hive-port
      DEFAULT_SCHEMA: default
      USERNAME: my-hive-username
      PASSWORD: my-hive-password
    volumes:
      - *shared-jdbc-drivers
      - ./output/hive-example:/output

PostgreSQL with basic authentication

Use <<: *psql under the environment section to use this connection type.

Available attributes:

• HOST - database host name or IP address (required)
• PORT - database port (optional, default is 5432)
• DATABASE - database name (required)
• USERNAME - database user name (required)
• PASSWORD - database user password (required)

Example

services:
  my-psql-database:
    <<: *extract
    environment:
      <<: *psql
      USERNAME: db-user
      PASSWORD: db-user-password
      HOST: psql-database-host.internal
      DATABASE: inventory
    volumes:
      - *shared-jdbc-drivers
      - ./output/my-psql-database:/output

PostgreSQL with IAM authentication

Use <<: *psql-iam under the environment section to use this connection type.

Available attributes:

• HOST - database host name or IP address (required)
• PORT - database port (optional, default is 5432)
• DATABASE - database name (required)
• USERNAME - database user name (required)
• AWS_ACCESS_KEY_ID - AWS access key id (required)
• AWS_SECRET_ACCESS_KEY - AWS secret access key (required)

Example

services:
  my-psql-database:
    <<: *extract
    environment:
      <<: *psql-iam
      AWS_ACCESS_KEY_ID: my-access-key-id
      AWS_SECRET_ACCESS_KEY: my-secret-access-key
      USERNAME: db-user
      HOST: psql-database-host.internal
      DATABASE: inventory
    volumes:
      - *shared-jdbc-drivers
      - ./output/my-psql-database:/output

MySQL with basic authentication

Use <<: *mysql under the environment section to use this connection type.

Available attributes:

• HOST - database host name or IP address (required)
• PORT - database port (optional, default is 3306)
• USERNAME - database user name (required)
• PASSWORD - database user password (required)

Example

services:
  my-mysql-database:
    <<: *extract
    environment:
      <<: *mysql
      USERNAME: db-user
      PASSWORD: db-user-password
      HOST: mysql-database-host.internal
    volumes:
      - *shared-jdbc-drivers
      - ./output/my-mysql-database:/output

MySQL with IAM authentication

Use <<: *mysql-iam under the environment section to use this connection type.

Available attributes:

• HOST - database host name or IP address (required)
• PORT - database port (optional, default is 3306)
• USERNAME - database user name (required)
• AWS_ACCESS_KEY_ID - AWS access key id (required)
• AWS_SECRET_ACCESS_KEY - AWS secret access key (required)

Example

services:
  my-mysql-database:
    <<: *extract
    environment:
      <<: *mysql-iam
      AWS_ACCESS_KEY_ID: my-access-key-id
      AWS_SECRET_ACCESS_KEY: my-secret-access-key
      USERNAME: db-user
      HOST: mysql-database-host.internal
    volumes:
      - *shared-jdbc-drivers
      - ./output/my-mysql-database:/output

Oracle with basic authentication

Use <<: *oracledb under the environment section to use this connection type.

Available attributes:

• HOST - database host name or IP address (required)
• PORT - database port (optional, default is 1521)
• USERNAME - database user name (required)
• PASSWORD - database user password (required)
• SERVICE - SID/service identifier (usually ORCL) or service name (required)
• DEFAULT_DATABASE_NAME - database name (usually same as SID) (required)

Example

services:
  my-oracledb-database:
    <<: *extract
    environment:
      <<: *oracledb
      HOST: oracle-host.internal
      USERNAME: db-user
      PASSWORD: db-user-password
      SERVICE: ORCL
      DEFAULT_DATABASE_NAME: ORCL
    volumes:
      - *shared-jdbc-drivers
      - ./output/my-oracledb-database:/output

Microsoft SQL Server with basic authentication

Use <<: *mssql under the environment section to use this connection type.

Available attributes:

• HOST - database host name or IP address (required)
• DATABASE - database name (required)
• USERNAME - database user name (required)
• PASSWORD - database user password (required)

Example

services:
  my-mssql-database:
    <<: *extract
    environment:
      <<: *mssql
      USERNAME: db-user
      PASSWORD: db-user-password
      HOST: mssql-database-host
      DATABASE: northwind
    volumes:
      - *shared-jdbc-drivers
      - ./output/mssql-database:/output