How to set up playbooks

A common question that data teams often face is how to automate metadata at scale.

Having started out as a data team ourselves, we know that automating repetitive tasks can help data teams maximize the value they provide to their organization. One way of doing so is through Atlan's playbooks!

Playbooks help power metadata automation for your data assets in Atlan. Users can create rule-based automations at scale and update metadata in bulk, helping streamline their workflows.

You can update the following asset metadata using playbooks:

• Certificates
• Descriptions
• Owners
• Terms
• Tags
• Domains
• Custom metadata

For example, imagine your organization needs to transfer ownership of several data assets. Instead of your data team manually updating the ownership of each and every asset, you could create a playbook to automate this process and update the metadata of your assets at scale.

Playbook recommendations

Before you begin, let's review some general guidelines on running playbooks in Atlan:

• Avoid running multiple playbooks simultaneously on the same set of assets. Allow one playbook run to be completed before proceeding with another operation on the same set of assets. Otherwise, you may experience performance issues and inconsistencies.
• Review and understand the depth of your asset lineage or hierarchy prior to enabling a tag propagation playbook. For assets with complex lineage, tag propagation may take longer to complete than the playbook runtime. You may want to review and judiciously select a list of assets that need to be tagged directly. For their child and/or downstream assets, Atlan recommends that you enable tag propagation.

Create a playbook

To create a playbook in Atlan:

1. From the left menu in Atlan, you can either:
   • Click Assets to navigate to the assets page.
     1. From the Filters menu on the left or the tabs along the top, apply any asset filters.
     2. Next to the search bar, click the 3-dot icon and then click Create playbook to create a playbook for the filtered assets — this option is only visible to admin users.
   • Click Governance to navigate to the governance center.
     1. Under the Governance heading of the Governance center, click Playbooks.
     2. Click Create New to get started.
2. In the Create new playbook dialog box, enter the following details:
   1. For Name, enter a name for the task to be accomplished — for example, Update ownership. (Atlan recommends that the length of a playbook name must be no longer than 46 characters.)
   2. (Optional) For Description, enter a description.
   3. (Optional) Select an icon for your playbook.
3. Click Create to save your playbook.

Set up rules as filters

To set up rules as filters for your playbook:

1. In the Build Rules page of your playbook, click Filters.
2. For name, add a name to your filter.
3. To set a matching condition for the filters, select Match all or Match any. Match all will logically AND the criteria, while Match any will logically OR the criteria.
4. For Attributes, select a relevant option:
   • For this example, we'll click Connection and then select a Snowflake connection. (Optional) To further refine your asset selection:
     1. Click All databases to filter by databases in a selected connection.
     2. Click All schemas to filter by schemas in a selected connection.
   • Click Connector to filter assets by supported connectors.
   • Click Asset type to filter by specific asset types — for example, tables, columns, queries, glossaries, and more.
   • Click Certificate to filter assets by certification status.
   • Click Owners to filter assets by asset owners.
   • Click Tags to filter assets by your tags in Atlan, including imported Snowflake and dbt tags.
     • (Optional) For Snowflake tags only, to the left of the checkbox, click Select value, and then from the Select tag value dialog, select any value(s) to filter assets by tag value.
   • Click Glossary, terms, & categories to filter by a specific glossary or category to bulk update all the nested terms or by multiple glossaries and categories.
   • Click Linked terms to filter assets by linked terms.
   • Click Domains to filter for data domains or subdomains. Note that Atlan currently does not support adding certificates, tags, and terms to domains and subdomains. 
   • Click Products to filter for data products by specific data domains or subdomains.
   • Click Schema qualified Name to filter assets by the qualified name of a given schema.
   • Click Database qualified Name to filter assets by the qualified name of a given database.
   • Click dbt to filter assets by dbt-specific filters and then select a dbt Cloud or dbt Core filter.
   • Click Properties to filter assets by common asset properties.
   • Click Usage to filter assets by usage metrics.
   • Click Monte Carlo to filter assets by Monte Carlo-specific filters.
   • Click Soda to filter assets by Soda-specific filters.
   • Click Table/View to filter tables or views by row count, column count, or size.
   • Click Column to filter columns by column-specific filters, including parent asset type or name, data type, or column keys.
   • Click Process to filter lineage processes by the SQL query.
   • Click Query to filter assets by associated visual queries.
   • Click Measure to filter Microsoft Power BI measures using the external measures filter.
5. For Operator, select Is one of for values to include or Is not for values to exclude. Depending on the selected attribute(s), you can also choose from additional operators:
   • Select Equals (=) or Not Equals (!=) to include or exclude assets through exact match search.
   • Select Starts With or Ends With to filter assets using the starting or ending sequence of values.
   • Select Contains or Does not contain to find assets with or without specified values contained within the attribute.
   • Select Pattern to filter assets using supported Elastic DSL regular expressions.
   • Select Is empty to filter assets with null values.
   • Select Belongs to or Doesn't belong to to filter data products by specific data domains or subdomains.
6. For Values, select the relevant values. The values will vary depending on the selected attributes.
7. (Optional) To add more filters, click Add filter and select Filter to add individual filters or Filter Group to nest more filters in a group.
8. (Optional) To view all the assets that match your rules, in the Filters card, click View all for a preview.

Select the actions

To select the actions to be performed based on your rules:

1. In the Build Rules page of your playbook, click Actions.
2. For Select Action, select the relevant metadata option to update:
   • Click Certificate to update the certification status of assets to Verified, Draft, Deprecated, or No certificate.
   • Click Description to update the description of your assets.
   • Click Owners to add, remove, or replace asset owners. In this example, we'll update the ownership of the assets.
   • Click Terms to add terms to your assets or remove or replace them from linked assets.
   • Click Tags to add tags to your assets or remove or replace them from tagged or propagated assets. Note that if there are multiple tag actions to be performed, Atlan will execute them in the following order: ADD, REMOVE, and then REPLACE.
   • Click Domain to add your assets to a specific domain or subdomain or remove them from an existing linked domain or subdomain.
   • Click any custom metadata structure and then select a custom metadata property to update or unlink it from your assets.
3. For Select operator, select the relevant option. The operators will vary depending on the selected action.
4. For Values, select the relevant option(s). The values will vary depending on the selected actions.
5. (Optional) To add more actions, click Add Action.

Run the playbook

If you'd like to continue working on your playbook, you can save it as a draft. If your playbook is ready, you can proceed to running it.

To run the playbook:

1. You can either:
   • To run the playbook once immediately, click Run once.
   • To schedule the playbook to run hourly, daily, weekly, or monthly, click Schedule and choose the preferred frequency, timezone, and time.
     🚨 Careful! If you're scheduling multiple playbooks, Atlan recommends spacing out the schedules as much as possible to minimize any overlap between the playbook workflow runs. For more about workflows in general, see workflow recommendations.
2. Click Complete to run the playbook.
3. In the resulting screen, click Go to profile to view your playbook profile.

Once your playbook has completed its run, you will see the metadata updated for your assets! 🎉