How to set up profiling playbooks

Monitoring and improving data quality is critical to building trust in your data assets. Atlan solves for this with profiling playbooks!

Profiling playbooks help power data observability for your assets in Atlan. You can create profiling playbooks to scan your assets at scale, identify any issues or inconsistencies, and improve the quality of your assets.

Supported sources

Atlan currently supports column profiling for the following connectors:

• Amazon Athena
• Amazon Redshift
• Google BigQuery
• Microsoft SQL Server
• MySQL
• PostgreSQL
• Snowflake
• Trino

Create a profiling playbook

To create a profiling playbook:

1. In the left menu in Atlan, click Governance.
2. Under the Governance heading of the Governance center, click Playbooks.
3. To the right of the Create New button, click the downward arrow and then select Profiling Playbook.
4. In the Create new profiling playbook dialog, enter the following details:
   1. For Name, enter a name for the task to be accomplished — for example, Tables scan.
   2. For Connection, select a supported connection from the dropdown menu — in this example, we'll select a Google BigQuery connection development.
   3. (Optional) For Description, enter a description for your playbook.
   4. (Optional) Select an icon for your playbook.
5. Click Create to save your playbook.

Set up rules as filters

To set up rules as filters for your profiling playbook:

1. In the Build Rules page of your profiling playbook, click Filters.
2. For the name field, add a name to your filter — for example, Profiling action.
3. To set a matching condition for the filters, select Match all or Match any. Match all will logically AND the criteria, while Match any will logically OR the criteria.
4. For Attributes, select the relevant option. For this example, we'll select Name listed under Properties.
5. For Operator, select Is one of for values to include or Is not for values to exclude.
6. For Values, select the relevant values. The values will vary depending on the selected attributes.
7. (Optional) To add more filters, click Add filter and select Filter to add individual filters or Filter Group to nest more filters in a group.
8. (Optional) To view all the assets that match your rules, in the Filters card, click View all for a preview.

Confirm profiling actions

The default profiling actions to be performed include:

• Base metrics:
  • Distinct count — number of rows that contain distinct values, relative to the column.
  • Missing count — number of rows that do not contain specific values.
• Numeric metrics:
  • Minimum and maximum values — smallest and greatest values in a numeric column.
  • Average — calculated average of values in a numeric column.
  • Standard deviation — calculated standard deviation of values in a numeric column.
  • Variance — calculated variance of values in a numeric column.
• String metrics:
  • Average length — average length of string values in a column.
  • Minimum and maximum length — minimum and maximum length of string values in a column.

Run the playbook

If you'd like to continue working on your playbook, you can save it as a draft. If your playbook is ready, you can proceed to running it.

To run the playbook:

1. You can either:
   • To run the playbook once immediately, click Run once.
   • To schedule the playbook to run hourly, daily, weekly, or monthly, click Schedule and choose the preferred frequency, timezone, and time.
2. Click Complete to confirm your selections.
3. In the resulting screen, click Go to profile to view your playbook profile.

Once your playbook run is completed, you will see the data profile updated for your assets! 🎉

View profiled assets

To view the profiled assets for your playbook:

1. In the Overview page of your playbook, to the right of Profiling action, click the total count of profiled assets.
2. In the sidebar to the right, profiled assets will be indicated with a bar graph icon. Click any profiled asset to proceed to viewing profiling data.
3. From the table sidebar, click the Column tab to view column assets and then select any of the profiled columns.
4. In the column sidebar to the right, click Profile to view profiling data for the selected column asset.