Secrets in Unity Catalog

A Unity Catalog secret is a securable object that stores a sensitive credential, such as a password, token, or API key. Secrets use the three-level namespace (catalog.schema.secret) and are available in every workspace attached to the metastore.

You govern secrets with the same Unity Catalog privileges as your other data assets, down to the individual secret. Users can browse a secret without being able to read or use it, you can encrypt secret values with your own key, and audit logs record every read or change.

Unity Catalog secrets can be Azure Databricks-managed or backed by Azure Key Vault. See Azure Key Vault secrets in Unity Catalog.

Requirements

• The workspace must be enabled for Unity Catalog. See What is Unity Catalog?.
• To read secrets with dbutils, use serverless notebooks and jobs on environment version 4 and above, or classic compute running Databricks Runtime 17.3 LTS and above. Azure Databricks recommends serverless or standard access mode.
• To use secrets in UDFs, see Use secrets in UDFs.

Create a secret

Secrets live in schemas. Create secrets in a schema that only the principals who need them can access. To decide how to group secrets into schemas, see Group secrets by who needs them.

A schema holds Azure Databricks-managed secrets or Azure Key Vault-backed secrets, not both, so you choose where values are stored when you set up the schema. For an Azure Key Vault-backed schema, see Create a secret in an Azure Key Vault-backed schema.

Create a Azure Databricks-managed secret

To create a Azure Databricks-managed secret, you need USE CATALOG on the catalog, and either ownership of the schema or CREATE SECRET and USE SCHEMA on it.

To create a schema for an application's secrets, run the following:

CREATE SCHEMA IF NOT EXISTS main.orders_app;

Catalog Explorer

1. In your Azure Databricks workspace, click Catalog to open Catalog Explorer.
2. Go to the schema where you want to create the secret.
3. Click Create > Secret.
4. Enter a name and value. Optionally, add a comment and an expiration date. If a secret expires, Catalog Explorer shows a warning.
5. Click Create.

REST API

To create a secret, run the following:

curl -X POST \
  -H "Authorization: Bearer $DATABRICKS_TOKEN" \
  -H 'Content-Type: application/json' \
  -d '{
    "catalog_name": "main",
    "schema_name": "orders_app",
    "name": "db_password",
    "value": "your_secret_value",
    "comment": "Password for the orders database"
  }' \
  "$DATABRICKS_HOST/api/2.1/unity-catalog/secrets"

Create a secret in an Azure Key Vault-backed schema

An admin backs a schema with Azure Key Vault one time. After that, you create, update, and rotate its secrets in Azure Key Vault, and they appear in the schema in Unity Catalog. To back a schema, see Azure Key Vault secrets in Unity Catalog.

1. Create the secret in the Azure Key Vault that backs the schema.
2. Read the secret by name. See Read a secret.

Grant access to secrets

Grant the privilege that matches what each principal needs to do. Except with BROWSE, every principal also needs USE CATALOG and USE SCHEMA on the parent catalog and schema. A privilege granted on a catalog or schema applies to the secrets in it, including secrets created later.

To grant privileges, use Catalog Explorer: open the secret or schema, click the Permissions tab, and click Grant. To grant with SQL, run the following:

-- Let a job's service principal read every secret in a schema
GRANT USE CATALOG ON CATALOG main TO `etl-service-principal`;
GRANT USE SCHEMA, READ SECRET ON SCHEMA main.orders_app TO `etl-service-principal`;

-- Let the application's developers create secrets in its schema
GRANT USE SCHEMA, CREATE SECRET ON SCHEMA main.orders_app TO `orders-developers`;

See Manage privileges in Unity Catalog.

Read a secret

To read a secret value, you need ownership of the secret, or READ SECRET on the secret or on its schema or catalog.

Secrets utility (dbutils.secrets)

Azure Databricks recommends dbutils, because it redacts the value in cell output and logs. The following example reads a database username and password to configure a JDBC read:

username = dbutils.secrets.get(catalog="main", schema="orders_app", key="db_username")
password = dbutils.secrets.get(catalog="main", schema="orders_app", key="db_password")

df = (spark.read
  .format("jdbc")
  .option("url", "<jdbc-url>")
  .option("dbtable", "<table-name>")
  .option("user", username)
  .option("password", password)
  .load()
)

See Secrets utility (dbutils.secrets).

REST API

To return the value, set include_value=true and read the effective_value field in the response:

curl -G \
  -H "Authorization: Bearer $DATABRICKS_TOKEN" \
  --data-urlencode "include_value=true" \
  "$DATABRICKS_HOST/api/2.1/unity-catalog/secrets/main.orders_app.db_password"

List, update, and delete secrets

To update a value, you need WRITE SECRET on the secret. To change its comment or expiration date, or to delete it, you need ownership of the secret or MANAGE on it, its schema, or its catalog.

For an Azure Key Vault-backed schema, update and delete secrets in Azure Key Vault. Deletes carry over to Unity Catalog.

Catalog Explorer

1. In Catalog Explorer, go to the schema and click Secrets in the Overview pane to list its secrets.
2. Click the secret you want to change.
3. In the top-right corner, click the kebab menu (vertical dots) and select Edit or Delete.
4. To edit, enter a new value or expiration date and click Confirm. To delete, enter the full name of the secret and click Delete.

Secrets utility (dbutils.secrets)

To list the secrets you can access in a schema, run the following:

all_secrets = dbutils.secrets.list(catalog="main", schema="orders_app")

REST API

To list secrets, run the following. If more results are available, pass the response's next_page_token as page_token:

curl -G \
  -H "Authorization: Bearer $DATABRICKS_TOKEN" \
  --data-urlencode "catalog_name=main" \
  --data-urlencode "schema_name=orders_app" \
  --data-urlencode "page_size=100" \
  "$DATABRICKS_HOST/api/2.1/unity-catalog/secrets"

To update a value, run the following. Only fields in both update_mask and the request body change:

curl -X PATCH \
  -H "Authorization: Bearer $DATABRICKS_TOKEN" \
  -H 'Content-Type: application/json' \
  -d '{"value": "new_secret_value"}' \
  "$DATABRICKS_HOST/api/2.1/unity-catalog/secrets/main.orders_app.db_password?update_mask=*"

To delete a secret, run the following:

curl -X DELETE \
  -H "Authorization: Bearer $DATABRICKS_TOKEN" \
  "$DATABRICKS_HOST/api/2.1/unity-catalog/secrets/main.orders_app.db_password"

Audit secret access

The system.access.audit system table records secret events. Reads appear with action_name = 'getSecret'. To list the last 7 days of secret events, run the following:

SELECT event_time, user_identity.email, action_name, request_params
FROM system.access.audit
WHERE service_name = 'unityCatalog'
  AND action_name LIKE '%Secret%'
  AND event_date >= current_date() - INTERVAL 7 DAYS
ORDER BY event_time DESC;

See Audit log system table reference.

Encrypt secret values with customer-managed keys

Azure Databricks encrypts secret values with Azure Databricks-managed keys by default. To use your own key, configure customer-managed key (CMK) settings on the catalog that holds the secrets. Secret values are stored in Unity Catalog, not in the catalog's storage location, so this works for any catalog, including one with an external storage location. For requirements and setup, see Customer-managed keys for Unity Catalog.

External secret values stay in your cloud secret manager, which encrypts them with its own keys. CMK settings on the catalog don't apply to them.

Use secrets in UDFs

A Unity Catalog Python or Scala UDF can use a secret value without giving callers access to the secret. Declare the secret in the UDF's SECRETS clause. The UDF reads it with the function owner's permissions:

• The owner needs READ SECRET on the secret to create or replace the UDF. If the owner loses READ SECRET, the UDF fails.
• Callers need only EXECUTE on the UDF and no privilege on the secret.

The UDF definition must set environment_version to 6 or above. On classic compute, use Databricks Runtime 19 and above in standard access mode. For examples, see Python UDFs and Scala UDFs.

Session-scoped UDFs, which use databricks.secrets.get() or com.databricks.Secrets.get(), read secrets with the caller's permissions, so every caller needs READ SECRET. See Python UDFs and Scala UDFs.

On dedicated access mode compute, you can't call these UDFs directly. You can call one from a Unity Catalog SQL function that you use as an attribute-based access control (ABAC) column mask, which Azure Databricks enforces on serverless compute. See Unsupported compute versions.

Limitations

• Quotas: A schema can hold up to 1,000 secrets, and a metastore up to 10,000.
• No global search or information schema: Secrets don't appear in global search or information schema tables. Use Catalog Explorer, dbutils, or the REST API.
• OAuth API scope: The Unity Catalog secrets API requires the unity-catalog OAuth API scope. The secrets scope applies only to workspace secret scopes.
• Init scripts: You can't use Unity Catalog secrets in global or cluster-scoped init scripts.
• Remote development: Databricks Connect and compiled JAR run modes can't read Unity Catalog secrets, because dbutils.secrets requires a Databricks Runtime-backed notebook or job.

Additional resources