Dedicated compute group access

Dedicated group compute in Azure Databricks runs notebooks and jobs using an assigned group's permissions. You can share the compute resource with group members and users who have permission to assume the group.

Dedicated group access mode allows users to get the operational efficiency of a standard access mode cluster while also securely supporting languages and workloads that are not supported by standard access mode, such as Databricks Runtime for ML, resilient distributed dataset (RDD) APIs, and R.

Group clusters are one way to assume a role in Azure Databricks. When a user connects to a group cluster, their user-identity permissions are replaced by the group's permissions for operations on that cluster. This is equivalent to assuming the group as a role for the duration of the cluster session. For other methods of assuming a role, including the role switcher, CLI, and API, see Switch RBAC roles.

Requirements

To use the dedicated group access mode:

  • The workspace must be enabled for Unity Catalog.
  • You must use Databricks Runtime 15.4 or above.
  • The group must be an account group that is assigned to the workspace.
  • The assigned group must have CAN MANAGE permissions on a workspace folder where they can keep notebooks, ML experiments, and other workspace artifacts used by the group cluster.

What is dedicated access mode?

Dedicated access mode, formerly single-user access mode, assigns a compute resource to one user or one group. A compute resource assigned to a group is called a group cluster. Only the assigned user, or users who can act as the assigned group, can run workloads on it.

Create a compute resource dedicated to a group

To create group compute, you need permission to use a compute policy or the unrestricted cluster creation entitlement. Unless you are a workspace admin, you must also be a member of the assigned group or have permission to assume it.

  1. In your Azure Databricks workspace, go to Compute and click Create compute.
  2. Expand the Advanced section.
  3. Under Access mode, click Manual and then select Dedicated (formerly: Single-user) from the dropdown menu.
  4. In the Single user or group field, select the group you want assigned to this resource.
  5. Configure the other desired compute settings, then click Create.

Grant access to group compute

To use a group cluster, you need both of the following:

  • The CAN ATTACH TO permission on the compute resource.
  • Membership in the assigned group, including through a nested group, or permission to assume the assigned group.

Assigning a group to a compute resource automatically grants that group CAN ATTACH TO. You can grant compute permissions to additional users or groups, but this does not change the assigned group or the workload's identity. Those users still need membership in, or permission to assume, the assigned group. Being a group manager alone does not grant permission to create, edit, delete, or use the cluster.

To grant compute access, you must have CAN MANAGE on the compute resource:

  1. In the sidebar, click Compute.
  2. On the compute resource's row, open the kebab menu Kebab menu icon. and select Edit permissions.
  3. In Permission Settings, click the Select user, group or service principal… drop-down menu and select the user or group.
  4. Select the CAN ATTACH TO permission from the permission drop-down menu.
  5. Click Add, then Save.

See Compute permissions.

To edit or permanently delete a group cluster, you need CAN MANAGE on the compute resource. Editing also requires membership in, or permission to assume, the assigned group, unless you are a workspace admin.

Use group compute from a notebook

To run a notebook with the assigned group's permissions:

  1. Open a notebook in a workspace folder that the assigned group can access.

  2. In the notebook's compute selector, select the group cluster. If it is not listed, click More… to find it.

  3. In a SQL cell, or a cell that starts with %sql, run the following command to check the session identity:

    SELECT session_user();

The result is the assigned group's name. Grant data and workspace permissions to that group for the operations the notebook performs. Your personal permissions do not add to the group's permissions on the cluster.

If you are already acting as a role through the workspace role switcher, that role must match the group assigned to the cluster. See Switch RBAC roles.

For more information about attaching notebooks, see Attach a notebook to an all-purpose compute resource.

Run jobs as a group

Jobs can run with a group's permissions, including when you configure the job through the API or SDK. In the job details, set Run as to the group assigned to the compute resource. The group needs permission to access the notebooks, files, and data used by the job.

In a Jobs API request, set run_as.group_name to the group's name:

{
  "run_as": {
    "group_name": "analytics-team"
  }
}

For the configuration steps, group requirements, and behavior when a task uses dedicated compute assigned to a different identity, see Set Run as to a group.

Best practices for managing group clusters

Because user permissions are scoped down to the group when using group clusters, Databricks recommends creating a /Workspace/Groups/<groupName> folder for each group you plan to use with a group cluster. Then, assign CAN MANAGE permissions on the folder to the group. This allows groups to avoid permission errors. All of the group's notebooks and workspace assets should be managed in the group folder.

You must also modify the following workloads to run on group clusters:

  • MLflow: Ensure you run the notebook from the group folder or run mlflow.set_tracking_uri("/Workspace/Groups/<groupName>").
  • AutoML: Set the optional experiment_dir parameter to “/Workspace/Groups/<groupName>” for your AutoML runs.
  • dbutils.notebook.run: Ensure the group has READ permission on the notebook being executed.

Permissioning behavior on group clusters

All commands, queries, and other actions performed on a group cluster use the permissions assigned to the group, not the individual user.

Individual user permissions cannot be enforced because all group members have full access to the Spark APIs and shared compute environment. If user-based permissions were applied, one member could query restricted data, and another member without access could still retrieve the results through the shared environment. Therefore, the group itself, not the user who is a member of the group, must have the necessary permissions to successfully perform the action.

For example, the group needs explicit permission to query a table, access a secret scope or secret, use a Unity Catalog connection credential, access a Git folder, or create a workspace object.

Example group permissions

When you create a data object using the group cluster, the group is assigned as the object's owner.

For example, if you have a notebook attached to a group cluster and run the following command:

use catalog main;
create schema group_cluster_group_schema;

Then run this query to check the owner of the schema:

describe schema group_cluster_group_schema;

Example description of group schema

Auditing group dedicated compute activity

There are two key identities involved when a group cluster runs a workload:

  1. The identity that initiates the workload on the group cluster
  2. The group whose permissions are used to perform the actual workload actions

The audit log system table records these identities under the following parameters:

  • identity_metadata.run_by: The authenticating identity that initiates the action.
  • identity_metadata.run_as: The authorizing group whose permissions are used for the action.

For interactive workloads, run_by identifies the user. For job workloads, it identifies the Jobs service's application service principal. See Set Run as to a group.

The following example query pulls up the identity metadata for an action taken with the group cluster:

select action_name, event_time, user_identity.email, identity_metadata
from system.access.audit
where user_identity.email = "uc-group-cluster-group" AND service_name = "unityCatalog"
order by event_time desc limit 100;

View the audit log system table reference for more example queries. See Audit log system table reference.

Audit logs delivered to customer storage do not include the authorizing group and authenticating identity as separate identity metadata fields. Use system.access.audit to view this metadata.

Known limitations

Dedicated group access has the following limitations:

  • Jobs that use Git fail because the temporary directory the job uses to check out the Git repo is not writable. Use Git folders instead.
  • The created_by field in the table and column lineage system tables records the group. These tables do not separately record the user who initiated the workload.
  • When attached to a group cluster, Catalog Explorer does not filter by assets only accessible to the group.
  • If a group is renamed, you must manually update any compute policies that reference the group name.
  • Group clusters are not supported in workspaces where access control lists (ACLs) are disabled (isWorkspaceAclsEnabled is false), because those workspaces lack security and data access controls.
  • The %run command and other actions in the notebook context use the user's permissions rather than the group's permissions. This is because these actions are handled by the notebook environment, not the cluster's environment. Alternative commands such as dbutils.notebook.run() are run on the cluster and therefore use the group's permissions.
  • The is_member(<group>) function returns false when invoked on a group cluster because the group is not a member of itself. To check membership across both group clusters and other access modes, use is_member(<group>) OR session_user() = <group>.
  • AI Search has restrictions when you act as a group. See supported operations and index restrictions.
  • File and folder deletion is not supported in group clusters.
  • The file upload UI does not support group clusters.
  • Last updated on