🧪 Preview feature! This feature is available for your experimentation, and we'd love your feedback. It may change before its final generally-available form. If you'd like to participate in the preview, reach out to your customer success manager for more information.
Once you have integrated Apache Spark/OpenLineage, you can use connector-specific filters for quick asset discovery. The following filters are currently supported:
- Status filter — last run status for an asset
- Duration filter — last run duration for an asset
Atlan maps the following assets and properties from Apache Spark/OpenLineage.
Jobs
Atlan maps jobs from Apache Spark to its SparkJob asset type.
| Source property | Atlan property |
|---|---|
appName |
sparkAppName |
master |
sparkMaster |
OpenLineage metadata
Atlan reports OpenLineage operational metadata for Spark jobs.
| Atlan property | Description |
|---|---|
sparkRunVersion |
Spark runtime version |
sparkRunOpenLineageVersion |
OpenLineage library version |
sparkRunStartTime |
job start time |
sparkRunEndTime |
job end time |
sparkRunOpenLineageState |
status of the job |
