How to set up Microsoft Azure Event Hubs

Atlan supports the following authentication methods for Microsoft Azure Event Hubs:

• SAS key — this method uses a connection string-primary key to fetch metadata.
• Service principal — in addition to a connection string-primary key, this method requires a client ID, client secret, and tenant ID to fetch metadata.

SAS key authentication

Create a shared access signature policy

You will need to create a shared access signature (SAS) policy in Microsoft Azure Event Hubs for authentication in Atlan.

The Manage permission is required for the following:

• Atlan requires read permissions of the configurations set to event hubs and the event hub namespace. Since Atlan currently only supports SAS policy-based authentication, Manage permission is required to provide this type of access. SAS policies do not support granular access control while Send or Listen permission is insufficient to crawl configuration metadata. Granular permissions will only be available once Atlan supports other authentication methods that allow for the granular access control capabilities of Microsoft Azure.
• To fetch the Azure Event Hub status attribute and Azure Event Hub consumer group assets through the Azure APIs.

To create a SAS policy for crawling Microsoft Azure Event Hubs:

1. Log in to the Azure portal.
2. Open the menu and search for or click Event Hubs.
3. On the Event Hubs page, click the namespace of your event hub. Copy your Event Hubs Namespace to use for authentication in Atlan.
4. In the left menu of your event hub namespace, under Settings, click Shared access policies.
5. On the Shared access policies page, click + Add to add a new SAS policy.
6. In the Add SAS policy sidebar, enter the following details:
   1. For Policy name, enter a meaningful name — for example, Atlan integration policy.
   2. To add the Manage permission to your SAS policy, click Manage.
   3. Click Create to finish setup.
7. On the Shared access policies page, select the newly created SAS policy.
8. From the corresponding SAS Policy dialog, under Connection string-primary key, click the clipboard icon to copy the connection string-primary key and store it in a secure location.

You will need your event hub namespace and the connection string-primary key for crawling Microsoft Azure Event Hubs.

Service principal authentication

You will need the following to authenticate the connection in Atlan:

• Connection string-primary key — required to crawl Kafka assets
• Client ID (application ID), client secret, and tenant ID (directory ID) — required to crawl Microsoft Azure Event Hubs assets 

Create a shared access signature policy

Follow the steps in Create a shared access signature policy to generate a connection string-primary key for crawling Microsoft Azure Event Hubs.

Register app with Microsoft Entra ID

You will need to register your service principal application with Microsoft Entra ID and note down the values of the tenant ID, client ID, and client secret.

To register your app with Microsoft Entra ID:

1. Log in to the Azure portal.
2. In the search bar, search for Microsoft Entra ID and select it from the dropdown list.
3. From the left menu of the Microsoft Entra ID page, click App registrations.
4. From the toolbar on the App registrations page, click + New registration.
5. On the Register an application page, for Name, enter a name for your service principal application and then click Register.
6. On the homepage of your newly created application, from the Overview screen, copy the values for the following fields and store them in a secure location:
   • Application (client) ID
   • Directory (tenant) ID
7. From the left menu of your newly created application page, click Certificates & secrets.
8. On the Certificates & secrets page, under Client secrets, click + New client secret.
9. In the Add a client secret screen, enter the following details:
   1. For Description, enter a description for your client secret.
   2. For Expiry, select when the client secret will expire.
   3. Click Add.
10. On the Certificates & secrets page, under Client secrets, for the newly created client secret, click the clipboard icon to copy the Value and store it in a secure location.

Add app to Event Hubs Data Sender role

You will need to add the service principal application created in the previous step to the Azure Event Hubs Data Sender role.

To add a service principal to the Azure Event Hubs Data Sender role:

1. Log in to the Azure portal.
2. Open the menu and search for or click Event Hubs.
3. On the Event Hubs page, click the namespace of your event hub.
4. From the left menu of your event hubs namespace page, click Access Control (IAM).
5. In the upper right of the Access Control (IAM) page, navigate to the Add a role assignment tile and then click Add.
6. On the Add a role assignment page, enter the following details:
   1. For Role, click the dropdown to select Azure Event Hubs Data Sender — this allows send access to Azure Event Hubs resources.
   2. For Assign access to, click the dropdown to select Azure AD user, group, or service principal.
   3. For Select, choose the service principal application you created in the previous step.
   4. Click Save to save your role assignment.