Atlan runs its crawlers through an orchestrated set of automated tasks.
To monitor these orchestrated set of tasks follow these steps.
Monitor the crawling process
You can visualize the individual tasks a workflow runs as a directed acyclic graph (or "DAG").
To visualize the crawling process:
- From the page that appears when running a workflow immediately, click the Monitor Run button.
- At any other moment:
- From the left menu, navigate to the Workflow Center.
- On the left of the screen, click on the tile containing the workflow you want to check.
- On the right of the screen, click the specific run of that workflow you want to check.
On the left of the screen under the Summary tab you can also see:
- The current status of the workflow run.
- The start and finish time of the workflow run.
- The elapsed time (duration) of the workflow run.
- Who triggered the workflow run and how (manually or automatically).
If a crawler fails due to an error, Atlan will show where the failure occurred in the visualization.
To review the failure of any workflow with an error:
- Open the workflow run visualization (using either option above).
- Under the Summary tab on the left of the screen, click the View Failed tasks button.
Atlan will take you to the Failed Tasks tab on the left of the screen. Here you can review details about the specific activity or activities that failed.
Review log files
Each task in the DAG may produce a log file containing additional details.
To review the log file for a specific activity:
- Click the task (activity node) in the DAG visualization.
- Open the Failed Tasks tab of a workflow run visualization (see steps above).
- To the right of each failed step, click the Logs button.
If there are any logs available, Atlan will display them on the screen.
💪 Did you know? Not every failed activity will produce a log. Look at the Message field of failed tasks for ideas about what went wrong when there is no log file available.