Databricks clusters should encrypt traffic between worker nodes

While a Spark job runs, worker nodes exchange intermediate data (shuffle data) over the cluster network. By default that traffic is not encrypted. Anyone with a foothold on the virtual network, or a misconfigured neighbouring workload, could read it. Microsoft documents a way to encrypt this traffic between nodes in the Databricks documentation, and Apache Spark has matching network and I/O encryption settings. This is a Level 2 control, meaning it is a stricter, security-focused setting that can cost some performance.

Sources: Microsoft Learn: encrypt traffic between cluster worker nodes, Apache Spark security.

In the Databricks workspace open Compute, select a cluster and choose Edit. Under Advanced options, look at the Spark tab for the encryption settings and at the Init scripts tab for the script that enables inter-node encryption. Cluster policies can enforce the same settings for every new cluster.

With the Databricks CLI, this read-only call prints one cluster's configuration, including its Spark settings and init scripts:

databricks clusters get --cluster-id 0000-000000-example

List all clusters first with databricks clusters list and compare each one against the settings in the Microsoft article. Try this on a non-production workspace to learn the output format.

Follow the Microsoft Learn article: create the init script it describes, store it in a location your workspace allows, attach it to the cluster, and add the Spark settings the article lists. Restart the cluster so the settings take effect. To cover all clusters, put the init script and settings in a cluster policy so users cannot create clusters without them.

Treat the change like any cluster configuration change: edit a test cluster first, run a representative job, then roll it out through the policy. ConfigCobra itself never changes your workspace. It reports the gap and gives you the steps.

Clusters must be restarted, so running jobs and interactive notebooks on them are interrupted. Encryption adds processor work on every worker, so shuffle-heavy jobs can run slower and may need slightly larger clusters to hold the same run time. Jobs that only process data on one node see little difference. Data engineers who create their own clusters will be affected if a cluster policy now forces the setting, so tell them before the policy goes live. Nothing changes for the people who read the results.