How to deploy a custom Apache Spark configuration in Azure Synapse Studio
0 reputation · 15 Feb 2022, 01:43 UTC
To deploy a custom Apache Spark configuration in Azure Synapse Studio, follow these steps:
- Open Synapse Studio, navigate to Manage > Apache Spark configurations.
- Click the New button to start a new configuration (or use Import to load a local .json/.txt/.conf file).
- Enter a valid Name, optional Description, and add any Annotations as needed.
- Under Configuration properties, click Add to insert key‑value pairs such as
spark.synapse.key1andspark.synapse.key2; if no property is added, the service uses default values. - Press Continue to validate the entered values.
- When validation succeeds, click Create to save the configuration.
- Finally, select Publish all to make the configuration available for notebooks, job definitions, or Spark pools.
Verification: after publishing, open a notebook or job definition, expand the Apache Spark configuration drop‑down, and confirm that the new name appears in the list.
Rollback: if the configuration causes issues, you can revert a pool, notebook, or job to the default configuration by editing the selection and choosing the built‑in default, or by deleting the custom configuration and republishing.