Azure Databricks should send its diagnostic logs to a log destination

Without delivered logs there is no record of who ran what in a Databricks workspace: sign-ins, cluster changes, job runs, permission changes and data access are all gone once you need them for an investigation. Diagnostic settings are what move those records out of the service and into a place with retention and search.

Microsoft explains the feature in Diagnostic settings in Azure Monitor, and the Databricks-specific log categories are described in the Azure Databricks documentation on diagnostic logs.

In the Azure portal open your Azure Databricks workspace resource, then Monitoring, then Diagnostic settings. You should see at least one setting with log categories ticked and a destination chosen.

This read-only Azure CLI command lists the settings on a workspace:

az monitor diagnostic-settings list \
  --resource "/subscriptions/SUB_ID/resourceGroups/RG/providers/Microsoft.Databricks/workspaces/WS_NAME" \
  --query "value[].{name:name, workspace:workspaceId, storage:storageAccountId, hub:eventHubName}"

An empty list means no logs are being delivered. Check the Microsoft Learn page above for the plan requirements of Databricks diagnostic logging.

In the portal: workspace, Diagnostic settings, Add diagnostic setting, name it, tick the log categories you need, choose a destination such as a Log Analytics workspace, and save.

With the CLI, first list the categories the workspace offers (read-only), then create the setting:

az monitor diagnostic-settings categories list \
  --resource "$WS_ID"

az monitor diagnostic-settings create \
  --name databricks-logs \
  --resource "$WS_ID" \
  --workspace "$LOG_ANALYTICS_ID" \
  --logs '[{"category":"accounts","enabled":true},{"category":"clusters","enabled":true},{"category":"jobs","enabled":true}]'

Use category names from the first command rather than the examples. ConfigCobra only reads this setting and never creates it for you.

Nothing changes for Databricks users or running jobs; the workspace behaves as before. The change lands on the people who run the destination: a Log Analytics workspace or storage account starts receiving data, which means ingestion and storage costs and a retention period that someone has to choose and own.

Security and platform teams gain a searchable history of workspace activity. Decide who may read these logs, because they describe user activity and can include sensitive names and paths.