Evaluators

Update an evaluator version

PATCH /evaluators/{name}/versions/{version}

Updates the specified evaluator version in place.

Query parameters

api-version string required

The API version to use for this operation.

Path parameters

name string required

The name of the resource

version string required

The version of the EvaluatorVersion to update.

Request Body

Evaluator resource

Content-Type: application/json

display_name string

Display Name for evaluator. It helps to find the evaluator easily in AI Foundry. It does not need to be unique.

metadata object

Metadata about the evaluator

[key: string] string

categories EvaluatorCategory[]

The categories of the evaluator

supported_evaluation_levels EvaluationLevel[]

Evaluation levels this evaluator supports (e.g., turn, conversation). When omitted on create, the service defaults to ["turn"]. On update, omitting this field leaves it unchanged; an empty list is rejected. Custom code-based evaluators support only turn; custom prompt-based evaluators support exactly one level (turn or conversation).

description string

The asset description text.

tags object

Tag dictionary. Tags can be added, removed, and updated.

[key: string] string

Responses

The request has succeeded.

Body

Content-Type: application/json

display_name string

Display Name for evaluator. It helps to find the evaluator easily in AI Foundry. It does not need to be unique.

metadata object

Metadata about the evaluator

[key: string] string

evaluator_type string | enum required

The type of the evaluator

Any of the following:

string

enum

builtincustom

categories EvaluatorCategory[] required

The categories of the evaluator

supported_evaluation_levels EvaluationLevel[]

definition code | prompt | rubric | endpoint required

Base evaluator configuration with discriminator

One of the following:

code CodeBasedEvaluatorDefinition

Code-based evaluator definition using python code

type enum required

code

code_text string

Inline code text for the evaluator

entry_point string

The entry point Python file name for the uploaded evaluator code (e.g. 'answer_length_evaluator.py')

image_tag string

The container image tag to use for evaluator code execution

blob_uri string

The blob URI for the evaluator storage

prompt PromptBasedEvaluatorDefinition

Prompt-based evaluator

type enum required

prompt

prompt_text string required

The prompt text used for evaluation

rubric RubricBasedEvaluatorDefinition

Rubric evaluator definition. Ordinary quality rubrics report one aggregate. Safety rubrics require Boolean scoring and an explicit allowed classification on every dimension, and report disallowed_behavior_violated and allowed_behavior_violated as two independent lower-is-better metrics. Safety rubrics have no weights, threshold, or combined safety score.

type enum required

rubric

dimensions Dimension[] required

The rubric criteria. Safety dimensions require allowed and omit weight. Rating generation includes a service residual; Boolean generation does not inject a generic residual. Existing ordinary and legacy definitions retain their behavior.

pass_threshold number

Optional normalized threshold for Likert scoring, interpreted as 0.5 when omitted. Ordinary Boolean scoring requires all applicable checks to pass: omit this field or set it to 1; its effective threshold is 1 and other values are rejected. Safety rubrics omit this field and report two separate metrics without an overall threshold.

scoring RubricScoring

Simple rubric scoring choices. Omission preserves existing legacy scoring.

mode string | enum

The judgment mode selected for an entire rubric. Safety rubrics use Boolean judgments with separately reported allowed and disallowed behavior metrics.

Any of the following:

string

enum

booleanlikert

endpoint EndpointBasedEvaluatorDefinition

Endpoint-based evaluator definition. The customer owns and hosts an HTTP endpoint that implements the evaluation contract. The evaluator references a Project Connection by name; the connection stores the endpoint URL and credentials (API Key or Entra ID). At execution time, the service resolves the connection to obtain the endpoint URL and authentication details, then calls the endpoint for each evaluation row.

type enum required

endpoint

connection_name string required

Name of the Project Connection that stores the endpoint URL and credentials. The connection must exist on the project and have a non-empty target URL. Supported auth types: ApiKey (sends api-key header) and AAD/Entra ID (acquires a bearer token via the project's Managed Identity).

schema_version string | enum

Contract version for an endpoint-based evaluator.

Any of the following:

string

enum

0.0.10.0.2

generation_artifacts EvaluatorGenerationArtifacts

Service-managed provenance artifacts produced by an evaluator generation job. Present only on EvaluatorVersion resources created via the generation pipeline. The combined-JSONL Foundry Dataset is read-only and resolves to a versioned dataset in a service-reserved namespace.

dataset VersionedDatasetReference required

Reference to a Dataset whose concrete version has already been resolved.

name string required

Dataset name.

version string required

Concrete Dataset version.

kinds string[] required

The kinds of rows present in dataset. Always contains "spec" (the generated evaluation specification, a Markdown document describing what the evaluator measures). May additionally contain "tools" (when the generation pipeline produced or inferred OpenAI tool schemas) and/or "context" (when supplementary materials such as file uploads or trace samples were used during generation).

generation_job_id string

Read-only provenance link back to the EvaluatorGenerationJob that produced this version. Present only on evaluator versions created via the generation pipeline; absent for manually-created versions and unaffected by subsequent PATCH calls.

warnings GenerationWarningType[]

Categories of warnings surfaced on this generated evaluator version. Present only on versions created via an EvaluatorGenerationJob when the paired job produced non-empty warnings. Absent (treat as no warnings) when the version is not from generation, when the paired job was clean, or when a subsequent PATCH to definition cleared the paired job's advisories. Follow generation_job_id to fetch the detailed warning payloads.

created_by string required

Creator of the evaluator

created_at string required

Creation date/time of the evaluator

modified_at string required

Last modified date/time of the evaluator

id string

Asset ID, a unique identifier for the asset

name string required

The name of the resource

version string required

The version of the resource

Update an evaluator version

Query parameters

Path parameters

Request Body

Responses

Body

Headers

Body

Headers

Body

Request

Response