Azure Databricks workspaces should encrypt sensitive data with a key you control

Azure Databricks always encrypts data at rest, but with a platform-managed key unless you say otherwise. A customer-managed key moves control of that key to you: you decide where it lives, who may use it and when it is rotated or revoked. That matters for workspaces that hold notebooks, query history, stored secrets, job results or model files that you would not want readable if the key were withdrawn.

Microsoft Learn describes four customer-managed key features for Azure Databricks, each for a different type of data: managed services (control plane data such as notebooks, queries and stored secrets), DBFS root (your workspace storage account), managed disks (temporary disk storage of cluster VMs) and Unity Catalog. Keys come from Azure Key Vault or Azure Key Vault Managed HSM. The feature needs the Premium plan, so a workspace on a lower tier cannot meet this setting without an upgrade.

Source: Customer-managed keys for encryption, Microsoft Learn.

In the Azure portal, open the Azure Databricks workspace, then Settings and Encryption. Each data type shows whether it uses a Microsoft-managed key or a customer-managed key, and which Key Vault key it points to.

From a terminal, this read-only Azure CLI call prints the encryption settings of one workspace (replace the placeholders):

az databricks workspace show \
  --resource-group <resource-group> \
  --name <workspace-name> \
  --query "{dbfsRoot: parameters.encryption.value, entities: encryption.entities}"

A key source of Default means a Microsoft-managed key; Microsoft.Keyvault means a customer-managed key. To review every workspace, run az databricks workspace list --query "[].{name:name, rg:resourceGroup}" and repeat the call. Run these on a test subscription first if you are unsure of your permissions.

Create or choose a key in Azure Key Vault (soft delete and purge protection should be on), then give the Azure Databricks service identity access to it. In the portal, open the workspace, go to Settings then Encryption, switch the data type to Use your own key and enter the Key Vault and key. Repeat for each data type you want covered.

A scripted example for DBFS root encryption (review it, then run it yourself; ConfigCobra never changes your tenant):

az databricks workspace update \
  --resource-group <resource-group> \
  --name <workspace-name> \
  --key-source Microsoft.Keyvault \
  --key-name <key-name> \
  --key-vault <https://your-vault.vault.azure.net> \
  --key-version <key-version>

Managed services and managed disks have their own steps; follow the matching Microsoft Learn pages linked from the source above.

Day-to-day use of notebooks, jobs and SQL does not change for analysts. The change lands on the people who run Key Vault and Azure Databricks: they now own the key lifecycle. If the key is deleted, disabled or its access is removed, the workspace can no longer read the data it protects, so key backups, soft delete, purge protection and a rotation process become part of operations.

Plan the change with the workspace owner and the Key Vault owner, pick a quiet period, and test on a non-production workspace first. Serverless compute is handled differently: Microsoft Learn notes that customer-managed keys for managed disks do not apply to serverless resources, because their disks are short-lived.