Evaluators
Create an evaluator version
Creates a new evaluator version with an auto-incremented version identifier.
Query parameters
api-version
string
required
The API version to use for this operation.
Path parameters
name
string
required
The name of the resource
Request Body
display_name
string
Display Name for evaluator. It helps to find the evaluator easily in AI Foundry. It does not need to be unique.
metadata
object
Metadata about the evaluator
evaluator_type
string | enum
required
The type of the evaluator
enum
categories
EvaluatorCategory[]
required
The categories of the evaluator
supported_evaluation_levels
EvaluationLevel[]
Evaluation levels this evaluator supports (e.g., turn, conversation). When omitted on create, the service defaults to ["turn"]. On update, omitting this field leaves it unchanged; an empty list is rejected. Custom code-based evaluators support only turn; custom prompt-based evaluators support exactly one level (turn or conversation).
definition
code | prompt | rubric | endpoint
required
Base evaluator configuration with discriminator
code
CodeBasedEvaluatorDefinition
Code-based evaluator definition using python code
type
enum
required
code_text
string
Inline code text for the evaluator
entry_point
string
The entry point Python file name for the uploaded evaluator code (e.g. 'answer_length_evaluator.py')
image_tag
string
The container image tag to use for evaluator code execution
blob_uri
string
The blob URI for the evaluator storage
prompt
PromptBasedEvaluatorDefinition
Prompt-based evaluator
type
enum
required
prompt_text
string
required
The prompt text used for evaluation
rubric
RubricBasedEvaluatorDefinition
Rubric evaluator definition. Ordinary quality rubrics report one aggregate. Safety rubrics require Boolean scoring and an explicit allowed classification on every dimension, and report disallowed_behavior_violated and allowed_behavior_violated as two independent lower-is-better metrics. Safety rubrics have no weights, threshold, or combined safety score.
type
enum
required
dimensions
Dimension[]
required
The rubric criteria. Safety dimensions require allowed and omit weight. Rating generation includes a service residual; Boolean generation does not inject a generic residual. Existing ordinary and legacy definitions retain their behavior.
pass_threshold
number
Optional normalized threshold for Likert scoring, interpreted as 0.5 when omitted. Ordinary Boolean scoring requires all applicable checks to pass: omit this field or set it to 1; its effective threshold is 1 and other values are rejected. Safety rubrics omit this field and report two separate metrics without an overall threshold.
scoring
RubricScoring
Simple rubric scoring choices. Omission preserves existing legacy scoring.
mode
string | enum
The judgment mode selected for an entire rubric. Safety rubrics use Boolean judgments with separately reported allowed and disallowed behavior metrics.
enum
endpoint
EndpointBasedEvaluatorDefinition
Endpoint-based evaluator definition. The customer owns and hosts an HTTP endpoint that implements the evaluation contract. The evaluator references a Project Connection by name; the connection stores the endpoint URL and credentials (API Key or Entra ID). At execution time, the service resolves the connection to obtain the endpoint URL and authentication details, then calls the endpoint for each evaluation row.
type
enum
required
connection_name
string
required
Name of the Project Connection that stores the endpoint URL and credentials. The connection must exist on the project and have a non-empty target URL. Supported auth types: ApiKey (sends api-key header) and AAD/Entra ID (acquires a bearer token via the project's Managed Identity).
schema_version
string | enum
Contract version for an endpoint-based evaluator.
enum
description
string
The asset description text.
tags
object
Tag dictionary. Tags can be added, removed, and updated.
Responses
The request has succeeded and a new resource has been created as a result.
Body
display_name
string
Display Name for evaluator. It helps to find the evaluator easily in AI Foundry. It does not need to be unique.
metadata
object
Metadata about the evaluator
evaluator_type
string | enum
required
The type of the evaluator
enum
categories
EvaluatorCategory[]
required
The categories of the evaluator
supported_evaluation_levels
EvaluationLevel[]
Evaluation levels this evaluator supports (e.g., turn, conversation). When omitted on create, the service defaults to ["turn"]. On update, omitting this field leaves it unchanged; an empty list is rejected. Custom code-based evaluators support only turn; custom prompt-based evaluators support exactly one level (turn or conversation).
definition
code | prompt | rubric | endpoint
required
Base evaluator configuration with discriminator
code
CodeBasedEvaluatorDefinition
Code-based evaluator definition using python code
type
enum
required
code_text
string
Inline code text for the evaluator
entry_point
string
The entry point Python file name for the uploaded evaluator code (e.g. 'answer_length_evaluator.py')
image_tag
string
The container image tag to use for evaluator code execution
blob_uri
string
The blob URI for the evaluator storage
prompt
PromptBasedEvaluatorDefinition
Prompt-based evaluator
type
enum
required
prompt_text
string
required
The prompt text used for evaluation
rubric
RubricBasedEvaluatorDefinition
Rubric evaluator definition. Ordinary quality rubrics report one aggregate. Safety rubrics require Boolean scoring and an explicit allowed classification on every dimension, and report disallowed_behavior_violated and allowed_behavior_violated as two independent lower-is-better metrics. Safety rubrics have no weights, threshold, or combined safety score.
type
enum
required
dimensions
Dimension[]
required
The rubric criteria. Safety dimensions require allowed and omit weight. Rating generation includes a service residual; Boolean generation does not inject a generic residual. Existing ordinary and legacy definitions retain their behavior.
pass_threshold
number
Optional normalized threshold for Likert scoring, interpreted as 0.5 when omitted. Ordinary Boolean scoring requires all applicable checks to pass: omit this field or set it to 1; its effective threshold is 1 and other values are rejected. Safety rubrics omit this field and report two separate metrics without an overall threshold.
scoring
RubricScoring
Simple rubric scoring choices. Omission preserves existing legacy scoring.
mode
string | enum
The judgment mode selected for an entire rubric. Safety rubrics use Boolean judgments with separately reported allowed and disallowed behavior metrics.
enum
endpoint
EndpointBasedEvaluatorDefinition
Endpoint-based evaluator definition. The customer owns and hosts an HTTP endpoint that implements the evaluation contract. The evaluator references a Project Connection by name; the connection stores the endpoint URL and credentials (API Key or Entra ID). At execution time, the service resolves the connection to obtain the endpoint URL and authentication details, then calls the endpoint for each evaluation row.
type
enum
required
connection_name
string
required
Name of the Project Connection that stores the endpoint URL and credentials. The connection must exist on the project and have a non-empty target URL. Supported auth types: ApiKey (sends api-key header) and AAD/Entra ID (acquires a bearer token via the project's Managed Identity).
schema_version
string | enum
Contract version for an endpoint-based evaluator.
enum
generation_artifacts
EvaluatorGenerationArtifacts
Service-managed provenance artifacts produced by an evaluator generation job. Present only on EvaluatorVersion resources created via the generation pipeline. The combined-JSONL Foundry Dataset is read-only and resolves to a versioned dataset in a service-reserved namespace.
dataset
VersionedDatasetReference
required
Reference to a Dataset whose concrete version has already been resolved.
name
string
required
Dataset name.
version
string
required
Concrete Dataset version.
kinds
string[]
required
The kinds of rows present in dataset. Always contains "spec" (the generated evaluation specification, a Markdown document describing what the evaluator measures). May additionally contain "tools" (when the generation pipeline produced or inferred OpenAI tool schemas) and/or "context" (when supplementary materials such as file uploads or trace samples were used during generation).
generation_job_id
string
Read-only provenance link back to the EvaluatorGenerationJob that produced this version. Present only on evaluator versions created via the generation pipeline; absent for manually-created versions and unaffected by subsequent PATCH calls.
warnings
GenerationWarningType[]
Categories of warnings surfaced on this generated evaluator version. Present only on versions created via an EvaluatorGenerationJob when the paired job produced non-empty warnings. Absent (treat as no warnings) when the version is not from generation, when the paired job was clean, or when a subsequent PATCH to definition cleared the paired job's advisories. Follow generation_job_id to fetch the detailed warning payloads.
created_by
string
required
Creator of the evaluator
created_at
string
required
Creation date/time of the evaluator
modified_at
string
required
Last modified date/time of the evaluator
id
string
Asset ID, a unique identifier for the asset
name
string
required
The name of the resource
version
string
required
The version of the resource
Request
POST /evaluators/{name}/versions HTTP/1.1Response
Status: 201
Content-Type: application/jsonStatus: 4XX
Content-Type: application/jsonStatus: 5XX
Content-Type: application/json