If an asset is deleted via API using the either delete asset by guid or the bulk endpoint, will workflows recreate the asset on next run? What is the expected behavior?
Atlan works on the principle of reflecting exactly what is in the source. Any deletes being executed at the source will results in assets within Atlan being archived (that is, soft deleted) as well.
Since our workflows are meant to execute in this sync, our recommendation for data sources that have connectors is generally that updates/deletes be managed entirely by the source workflows.
Considering our workflows also run differential crawls, that is, any delta change on assets is what's crawled and updated in Atlan (this helps deliver faster runtimes and performant workflows), the following it what will happen if any explicit delete is done using APIs:
Archiving (soft delete)
- The asset moves to an Archived state
- The next workflow run will not unarchive/restore the asset automatically if the source asset does not undergo a change to one of the source metadata properties
- The next workflow run will unarchive/restore the asset in its complete form, if one of the source properties changes (for example, a new column was added, and so on)
Using the restore endpoint on the asset will unarchive the asset as well.
Hard delete
- The asset is completely removed from the metastore
- The next workflow run will not unarchive/restore the asset automatically if the source asset does not undergo a change to one of the source metadata properties
- The next workflow run will unarchive/restore the asset in a partial form, if one of the source properties changes (for example, a new column was added, and so on)
- What does partial form mean? The asset will come back with only that one property/key present. For example, if a new column is added, the table will only have that one column - since all other columns were hard deleted. This will lead to the asset being incomplete and showing an incorrect representation of the source.
Using the restore endpoint will not unarchive the asset.
Hence, our recommendation for this to be managed by workflows. We provisioned the delete and restore endpoints for assets created via APIs (that is, not being crawled via a workflow).
