Evaluation Rules
Create or update an evaluation rule
Creates a new evaluation rule, or replaces the existing rule when the identifier matches.
Query parameters
api-version
string
required
The API version to use for this operation.
Path parameters
id
string
required
Unique identifier for the evaluation rule.
Header parameters
Foundry-Features
enum
A feature flag opt-in required when using preview operations or modifying persisted preview resources.
Request Body
Evaluation rule resource
id
string
required
Unique identifier for the evaluation rule.
displayName
string
Display Name for the evaluation rule.
description
string
Description for the evaluation rule.
action
continuousEvaluation | traceEvaluation | datasetEvaluation | humanEvaluationPreview
required
Evaluation action model.
continuousEvaluation
ContinuousEvaluationRuleAction
Evaluation rule action for continuous evaluation.
type
enum
required
evalId
string
required
Eval Id to add continuous evaluation runs to.
maxHourlyRuns
integer
Maximum number of evaluation runs allowed per hour.
samplingRate
number
Percentage (0-100] chance that a matching event triggers an evaluation. When omitted, the service-default is to evaluate every event, which is equivalent to setting a sampling rate of 100.
traceEvaluation
TraceEvaluationRuleAction
Configures a scheduled evaluation of observed agent traces.
type
enum
required
evalId
string
required
Identifier of the evaluation used to score the selected traces.
source
TracesEvalRunDataSource | TraceSourceEvalRunDataSource | TracesPreviewEvalRunDataSource | TraceSourcePreviewEvalRunDataSource
required
Existing trace-evaluation inputs supported by scheduled rules.
type
enum
required
The type of data source, always azure_ai_traces.
trace_ids
string[]
Collection of Agent trace identifiers that should be evaluated.
conversation_ids
string[]
Collection of conversation identifiers to filter traces by conversation.
agent_id
string
The agent ID used to filter traces for evaluation.
agent_name
string
The agent name used to filter traces for evaluation.
lookback_hours
integer
Lookback window (in hours) applied when retrieving traces from Application Insights. For scheduled evaluations this is inferred from the recurrence interval.
max_traces
integer
Sampling limit applied to traces retrieved for evaluation.
ingestion_delay_seconds
integer
The delay to apply for ingestion when querying traces.
type
enum
required
The type of data source, always azure_ai_trace_data_source.
trace_source
trace_id_source | conversation_id_source | agent_filter
Base class for polymorphic trace sources nested within a TraceSourceEvalRunDataSource.
trace_id_source
TraceIdTraceSource
A trace source that selects traces by explicit trace IDs.
type
enum
required
The type of trace source, always trace_id_source.
trace_ids
string[]
required
Collection of trace identifiers to evaluate.
conversation_id_source
ConversationIdTraceSource
A trace source that selects traces by conversation IDs.
type
enum
required
The type of trace source, always conversation_id_source.
conversation_ids
string[]
required
Collection of conversation identifiers to filter traces by.
agent_filter
AgentFilterTraceSource
A trace source that selects traces by agent reference with time-based filtering.
type
enum
required
The type of trace source, always agent_filter.
agent_id
string
The identifier of the agent to filter traces by.
agent_name
string
The name of the agent to filter traces by.
agent_version
string
The version of the agent to filter traces by.
max_traces
integer
Maximum number of traces to sample. Must be greater than 0.
filter_strategy
string | enum
Specifies the type of filter strategy used to select traces for evaluation.
enum
type
enum
required
The type of data source, always azure_ai_traces_preview.
trace_ids
string[]
Collection of Agent trace identifiers that should be evaluated.
conversation_ids
string[]
Collection of conversation identifiers to filter traces by conversation.
agent_id
string
The agent ID used to filter traces for evaluation.
agent_name
string
The agent name used to filter traces for evaluation.
lookback_hours
integer
Lookback window (in hours) applied when retrieving traces from Application Insights. For scheduled evaluations this is inferred from the recurrence interval.
max_traces
integer
Sampling limit applied to traces retrieved for evaluation.
ingestion_delay_seconds
integer
The delay to apply for ingestion when querying traces.
type
enum
required
The type of data source, always azure_ai_trace_data_source_preview.
trace_source
trace_id_source | conversation_id_source | agent_filter
Base class for polymorphic trace sources nested within a TraceSourceEvalRunDataSource.
trace_id_source
TraceIdTraceSource
A trace source that selects traces by explicit trace IDs.
type
enum
required
The type of trace source, always trace_id_source.
trace_ids
string[]
required
Collection of trace identifiers to evaluate.
conversation_id_source
ConversationIdTraceSource
A trace source that selects traces by conversation IDs.
type
enum
required
The type of trace source, always conversation_id_source.
conversation_ids
string[]
required
Collection of conversation identifiers to filter traces by.
agent_filter
AgentFilterTraceSource
A trace source that selects traces by agent reference with time-based filtering.
type
enum
required
The type of trace source, always agent_filter.
agent_id
string
The identifier of the agent to filter traces by.
agent_name
string
The name of the agent to filter traces by.
agent_version
string
The version of the agent to filter traces by.
max_traces
integer
Maximum number of traces to sample. Must be greater than 0.
filter_strategy
string | enum
Specifies the type of filter strategy used to select traces for evaluation.
enum
evaluation_level
enum | string
The level at which evaluation is performed.
enum
datasetEvaluation
DatasetEvaluationRuleAction
Configures a scheduled evaluation of dataset inputs.
type
enum
required
evalId
string
required
Identifier of the evaluation used to score the dataset.
source
OpenAI.CreateEvalJsonlRunDataSource | CsvEvalRunDataSource | OpenAI.CreateEvalCompletionsRunDataSource | TargetCompletionEvalRunDataSource | UserConversationSimulationEvalRunDataSource | UserConversationSimulationPreviewEvalRunDataSource
required
Dataset inputs supported by scheduled rules. Targets are configured within the selected source.
type
enum
required
The type of data source. Always jsonl.
source
OpenAI.EvalJsonlFileContentSource | OpenAI.EvalJsonlFileIdSource
required
Determines what populates the item namespace in the data source.
type
enum
required
The type of jsonl source. Always file_content.
content
OpenAI.EvalJsonlFileContentSourceContent[]
required
The content of the jsonl file.
type
enum
required
The type of jsonl source. Always file_id.
id
string
required
The identifier of the file.
type
enum
required
The type of data source, always csv.
source
EvalCsvFileIdSource
required
Represents a reference to an uploaded CSV file used as a source for evaluation data.
type
enum
required
The type of source, always file_id.
id
string
required
The identifier of the uploaded CSV file.
type
enum
required
The type of run data source. Always completions.
input_messages
OpenAI.CreateEvalCompletionsRunDataSourceInputMessagesTemplate | OpenAI.CreateEvalCompletionsRunDataSourceInputMessagesItemReference
Used when sampling from a model. Dictates the structure of the messages passed into the model. Can either be a reference to a prebuilt trajectory (ie, item.input_trajectory), or a template with variable references to the item namespace.
type
enum
required
type
enum
required
sampling_params
OpenAI.CreateEvalCompletionsRunDataSourceSamplingParams
reasoning_effort
string | enum | null
Constrains effort on reasoning for reasoning models. Currently supported values are none, minimal, low, medium, high, xhigh, and max. Reducing reasoning effort can result in faster responses and fewer tokens used on reasoning in a response. Not all reasoning models support every value. See the reasoning guide for model-specific support.
enum
response_format
OpenAI.ResponseFormatText | OpenAI.ResponseFormatJsonSchema | OpenAI.ResponseFormatJsonObject
type
enum
required
The type of response format being defined. Always text.
type
enum
required
The type of response format being defined. Always json_schema.
json_schema
OpenAI.ResponseFormatJsonSchemaJsonSchema
required
schema
OpenAI.ResponseFormatJsonSchemaSchema
The schema for the response format, described as a JSON Schema object. Learn how to build JSON schemas here.
type
enum
required
The type of response format being defined. Always json_object.
model
string
The name of the model to use for generating completions (e.g. "o3-mini").
source
OpenAI.EvalJsonlFileContentSource | OpenAI.EvalJsonlFileIdSource | OpenAI.EvalStoredCompletionsSource
required
Determines what populates the item namespace in this run's data source.
type
enum
required
The type of jsonl source. Always file_content.
content
OpenAI.EvalJsonlFileContentSourceContent[]
required
The content of the jsonl file.
type
enum
required
The type of jsonl source. Always file_id.
id
string
required
The identifier of the file.
type
enum
required
The type of source. Always stored_completions.
metadata
OpenAI.Metadata | null
Set of 16 key-value pairs that can be attached to an object. This can be useful for storing additional information about the object in a structured format, and querying for objects via API or the dashboard. Keys are strings with a maximum length of 64 characters. Values are strings with a maximum length of 512 characters.
type
enum
required
The type of data source, always azure_ai_target_completions.
input_messages
OpenAI.CreateEvalCompletionsRunDataSourceInputMessagesItemReference
type
enum
required
source
OpenAI.EvalJsonlFileContentSource | OpenAI.EvalJsonlFileIdSource
required
The source configuration for inline or file data.
type
enum
required
The type of jsonl source. Always file_content.
content
OpenAI.EvalJsonlFileContentSourceContent[]
required
The content of the jsonl file.
type
enum
required
The type of jsonl source. Always file_id.
id
string
required
The identifier of the file.
target
azure_ai_model | azure_ai_agent
required
Base class for targets with discriminator support.
azure_ai_model
AzureAIModelTarget
A target that identifies an Azure AI model. See evaluate administrator-connected models.
type
enum
required
The type of target, always azure_ai_model.
model
string
The name of the model deployment to invoke as the evaluation target.
sampling_params
ModelSamplingParams
Represents a set of parameters used to control the sampling behavior of a language model during text generation.
temperature
number
The temperature parameter for sampling. Defaults to 1.0.
top_p
number
The top-p parameter for nucleus sampling. Defaults to 1.0.
seed
integer
The random seed for reproducibility. Defaults to 42.
max_completion_tokens
integer
The maximum number of tokens allowed in the completion.
azure_ai_agent
AzureAIAgentTarget
A target that identifies an Azure AI agent. See evaluate model or agent targets.
type
enum
required
The type of target, always azure_ai_agent.
name
string
required
The name of the Azure AI agent to invoke as the evaluation target.
version
string
The version of the Azure AI agent. When omitted, the latest version is used.
tool_descriptions
ToolDescription[]
Optional descriptions of tools available to the agent.
tools
OpenAI.Tool[]
Tool definitions made available when invoking the agent.
item_generation_params
red_team | red_team_seed_prompts | red_team_taxonomy | response_retrieval | target_completion | synthetic_data_gen
Configures how the service prepares input items for an evaluation run before applying testing criteria. Depending on type, preparation can generate new inputs, retrieve existing responses, transform source data, or simulate conversations. Each type selects a workflow-specific model with the parameters required by that preparation workflow. These parameters do not define the evaluated target or the testing criteria.
red_team
RedTeamItemGenerationParams
Represents the parameters for red team item generation.
type
enum
required
The type of item generation parameters.
attack_strategies
AttackStrategy[]
required
The collection of attack strategies to be used.
num_turns
integer
required
The number of turns allowed in the game.
red_team_seed_prompts
RedTeamSeedPromptsItemGenerationParams
Represents the parameters for red team seed prompts item generation.
type
enum
required
The type of item generation parameters, always red_team_seed_prompts.
attack_strategies
AttackStrategy[]
required
The collection of attack strategies to be used.
num_turns
integer
required
The number of turns allowed in the game.
source
OpenAI.EvalJsonlFileContentSource
required
type
enum
required
The type of jsonl source. Always file_content.
content
OpenAI.EvalJsonlFileContentSourceContent[]
required
The content of the jsonl file.
red_team_taxonomy
RedTeamTaxonomyItemGenerationParams
Represents the parameters for red team taxonomy item generation.
type
enum
required
The type of item generation parameters, always red_team_taxonomy.
attack_strategies
AttackStrategy[]
required
The collection of attack strategies to be used.
num_turns
integer
required
The number of turns allowed in the game.
source
OpenAI.EvalJsonlFileContentSource
required
type
enum
required
The type of jsonl source. Always file_content.
content
OpenAI.EvalJsonlFileContentSourceContent[]
required
The content of the jsonl file.
response_retrieval
ResponseRetrievalItemGenerationParams
Parameters for retrieving deployed conversations for evaluation. See evaluate deployed conversations.
type
enum
required
The type of item generation parameters, always response_retrieval.
max_num_turns
integer
required
The maximum number of turns of chat history to evaluate.
data_mapping
object
required
Maps response_id to the source field that contains a stored Responses API response identifier. For example, {"response_id": "{{item.resp_id}}"}. Field names are case-sensitive.
source
OpenAI.EvalJsonlFileContentSource | OpenAI.EvalJsonlFileIdSource
required
The inline JSONL content containing the response identifiers to retrieve. The azure_ai_responses data source supports file_content only; providing a file_id results in a 400 Bad Request response.
type
enum
required
The type of jsonl source. Always file_content.
content
OpenAI.EvalJsonlFileContentSourceContent[]
required
The content of the jsonl file.
type
enum
required
The type of jsonl source. Always file_id.
id
string
required
The identifier of the file.
target_completion
TargetCompletionItemGenerationParams
Parameters for generating target completions for simulated conversations. See simulate conversations.
type
enum
required
The type of item generation parameters, always target_completion.
max_concurrency
integer
The maximum number of target completions to execute concurrently. The services defaults to 1 if a value is not specified by the caller, which invokes the target sequentially.
synthetic_data_gen
SyntheticDataGenerationItemGenerationParams
Represents the parameters for synthetic data generation item generation.
type
enum
required
The type of item generation parameters.
samples_count
integer
required
The maximum number of data samples to generate.
prompt
string
The prompt used to generate synthetic data. This property is optional when the target is an agent that already has instructions.
model_deployment_name
string
required
The name of the model deployment to use for generating synthetic data.
output_dataset_name
string
The name of the output dataset where generated synthetic data will be stored. If not provided, service generates dataset name automatically.
output_dataset_id
string
The identifier of the output dataset where generated synthetic data is stored. The generated data is a jsonl file with columns id, query and test_description.
sources
OpenAI.EvalJsonlFileIdSource[]
required
The optional seed data content source files for data generation.
type
enum
required
The type of data source, always azure_ai_user_conversation_simulation.
source
OpenAI.EvalJsonlFileContentSource | OpenAI.EvalJsonlFileIdSource | UserConversationSimulationInlineTestCasesSource
required
Sources supported by user conversation simulation.
type
enum
required
The type of jsonl source. Always file_content.
content
OpenAI.EvalJsonlFileContentSourceContent[]
required
The content of the jsonl file.
type
enum
required
The type of jsonl source. Always file_id.
id
string
required
The identifier of the file.
type
enum
required
The source type. Always inline_user_conversation_simulation.
test_cases
UserConversationSimulationTestCase[]
required
Test scenarios from which user conversations are simulated. At least one test case is required.
target
azure_ai_model | azure_ai_agent
required
Base class for targets with discriminator support.
azure_ai_model
AzureAIModelTarget
A target that identifies an Azure AI model. See evaluate administrator-connected models.
type
enum
required
The type of target, always azure_ai_model.
model
string
The name of the model deployment to invoke as the evaluation target.
sampling_params
ModelSamplingParams
Represents a set of parameters used to control the sampling behavior of a language model during text generation.
temperature
number
The temperature parameter for sampling. Defaults to 1.0.
top_p
number
The top-p parameter for nucleus sampling. Defaults to 1.0.
seed
integer
The random seed for reproducibility. Defaults to 42.
max_completion_tokens
integer
The maximum number of tokens allowed in the completion.
azure_ai_agent
AzureAIAgentTarget
A target that identifies an Azure AI agent. See evaluate model or agent targets.
type
enum
required
The type of target, always azure_ai_agent.
name
string
required
The name of the Azure AI agent to invoke as the evaluation target.
version
string
The version of the Azure AI agent. When omitted, the latest version is used.
tool_descriptions
ToolDescription[]
Optional descriptions of tools available to the agent.
tools
OpenAI.Tool[]
Tool definitions made available when invoking the agent.
model_configuration
EvaluationModelConfiguration
required
Configures the model that plays the simulated user. This model is separate from the evaluated target.
model
string
required
Model deployment that generates simulated user turns, in the format {connectionName}/modelDeploymentName.
sampling_params
ModelSamplingParams
Represents a set of parameters used to control the sampling behavior of a language model during text generation.
temperature
number
The temperature parameter for sampling. Defaults to 1.0.
top_p
number
The top-p parameter for nucleus sampling. Defaults to 1.0.
seed
integer
The random seed for reproducibility. Defaults to 42.
max_completion_tokens
integer
The maximum number of tokens allowed in the completion.
voice_model
azure-standard
Voice configuration used to convert text to speech for evaluation through the Voice Live endpoint.
azure-standard
EvaluationAzureStandardVoiceModelConfiguration
Configures an Azure standard neural voice used to convert text to speech for evaluation through the Voice Live endpoint.
type
enum
required
The voice kind. Always azure-standard.
name
string
required
The Azure neural voice name.
temperature
number
The synthesis temperature, from 0 to 1. When omitted, the service defaults to the underlying voice model's default temperature.
default_simulation_configuration
UserConversationSimulationDefaultConfiguration
Configures default conversation behavior and optional run-wide persistence of generated conversations.
max_num_turns
integer
Hard limit on turns in each conversation. When omitted, the service defaults to 20.
conversation_repetitions
integer
Number of independent conversation repetitions for each test case. Defaults to 1 when not specified at either the data-source or test-case level.
desired_num_turns
integer
Target number of turns in each conversation. The effective value cannot exceed the effective max_num_turns. When omitted, no target is set and the simulation model determines the conversation length dynamically from the scenario.
audio_effects
UserConversationSimulationAudioEffectsConfiguration
Configures effects applied to simulated conversation audio.
effects
UserConversationSimulationAudioEffect[]
Effects to apply to simulated audio.
volume_percentage
integer
Volume of the configured audio effects, as a percentage from 1 through 100. When omitted, the service defaults to 15.
user_behavior
UserConversationSimulationUserBehaviorConfiguration
Configures conversation behavior for the simulated user.
interruption
default
Configuration for simulated user interruption behavior.
default
UserConversationSimulationDefaultInterruptionConfiguration
Configures the default interruption behavior for the simulated user.
type
enum
required
The interruption type, always default.
enable_conversation_dataset_generation
boolean
Whether to write all generated conversations to a versioned Foundry dataset. When enabled, the completed run includes a simulated_user_conversations entry in output_datasets. When omitted, the service defaults to false.
output_conversation_dataset_name
string
Name of the dataset to which generated conversations are written. This property is used only when enable_conversation_dataset_generation is true. When omitted, the service generates a dataset name.
data_mapping
object
Maps JSONL dataset attributes to user conversation simulation inputs when their names differ from the defaults. The default attribute names are test_case_id, test_case_category, test_case_description, and simulation_configuration, matching UserConversationSimulationTestCase. Omit when the dataset uses these names or when using inline_user_conversation_simulation.
type
enum
required
The type of data source, always azure_ai_user_conversation_simulation_preview.
source
OpenAI.EvalJsonlFileContentSource | OpenAI.EvalJsonlFileIdSource | UserConversationSimulationInlineTestCasesSource
required
Sources supported by user conversation simulation.
type
enum
required
The type of jsonl source. Always file_content.
content
OpenAI.EvalJsonlFileContentSourceContent[]
required
The content of the jsonl file.
type
enum
required
The type of jsonl source. Always file_id.
id
string
required
The identifier of the file.
type
enum
required
The source type. Always inline_user_conversation_simulation.
test_cases
UserConversationSimulationTestCase[]
required
Test scenarios from which user conversations are simulated. At least one test case is required.
target
azure_ai_model | azure_ai_agent
required
Base class for targets with discriminator support.
azure_ai_model
AzureAIModelTarget
A target that identifies an Azure AI model. See evaluate administrator-connected models.
type
enum
required
The type of target, always azure_ai_model.
model
string
The name of the model deployment to invoke as the evaluation target.
sampling_params
ModelSamplingParams
Represents a set of parameters used to control the sampling behavior of a language model during text generation.
temperature
number
The temperature parameter for sampling. Defaults to 1.0.
top_p
number
The top-p parameter for nucleus sampling. Defaults to 1.0.
seed
integer
The random seed for reproducibility. Defaults to 42.
max_completion_tokens
integer
The maximum number of tokens allowed in the completion.
azure_ai_agent
AzureAIAgentTarget
A target that identifies an Azure AI agent. See evaluate model or agent targets.
type
enum
required
The type of target, always azure_ai_agent.
name
string
required
The name of the Azure AI agent to invoke as the evaluation target.
version
string
The version of the Azure AI agent. When omitted, the latest version is used.
tool_descriptions
ToolDescription[]
Optional descriptions of tools available to the agent.
tools
OpenAI.Tool[]
Tool definitions made available when invoking the agent.
model_configuration
EvaluationModelConfiguration
required
Configures the model that plays the simulated user. This model is separate from the evaluated target.
model
string
required
Model deployment that generates simulated user turns, in the format {connectionName}/modelDeploymentName.
sampling_params
ModelSamplingParams
Represents a set of parameters used to control the sampling behavior of a language model during text generation.
temperature
number
The temperature parameter for sampling. Defaults to 1.0.
top_p
number
The top-p parameter for nucleus sampling. Defaults to 1.0.
seed
integer
The random seed for reproducibility. Defaults to 42.
max_completion_tokens
integer
The maximum number of tokens allowed in the completion.
voice_model
azure-standard
Voice configuration used to convert text to speech for evaluation through the Voice Live endpoint.
azure-standard
EvaluationAzureStandardVoiceModelConfiguration
Configures an Azure standard neural voice used to convert text to speech for evaluation through the Voice Live endpoint.
type
enum
required
The voice kind. Always azure-standard.
name
string
required
The Azure neural voice name.
temperature
number
The synthesis temperature, from 0 to 1. When omitted, the service defaults to the underlying voice model's default temperature.
default_simulation_configuration
UserConversationSimulationDefaultConfiguration
Configures default conversation behavior and optional run-wide persistence of generated conversations.
max_num_turns
integer
Hard limit on turns in each conversation. When omitted, the service defaults to 20.
conversation_repetitions
integer
Number of independent conversation repetitions for each test case. Defaults to 1 when not specified at either the data-source or test-case level.
desired_num_turns
integer
Target number of turns in each conversation. The effective value cannot exceed the effective max_num_turns. When omitted, no target is set and the simulation model determines the conversation length dynamically from the scenario.
audio_effects
UserConversationSimulationAudioEffectsConfiguration
Configures effects applied to simulated conversation audio.
effects
UserConversationSimulationAudioEffect[]
Effects to apply to simulated audio.
volume_percentage
integer
Volume of the configured audio effects, as a percentage from 1 through 100. When omitted, the service defaults to 15.
user_behavior
UserConversationSimulationUserBehaviorConfiguration
Configures conversation behavior for the simulated user.
interruption
default
Configuration for simulated user interruption behavior.
default
UserConversationSimulationDefaultInterruptionConfiguration
Configures the default interruption behavior for the simulated user.
type
enum
required
The interruption type, always default.
enable_conversation_dataset_generation
boolean
Whether to write all generated conversations to a versioned Foundry dataset. When enabled, the completed run includes a simulated_user_conversations entry in output_datasets. When omitted, the service defaults to false.
output_conversation_dataset_name
string
Name of the dataset to which generated conversations are written. This property is used only when enable_conversation_dataset_generation is true. When omitted, the service generates a dataset name.
data_mapping
object
Maps JSONL dataset attributes to user conversation simulation inputs when their names differ from the defaults. The default attribute names are test_case_id, test_case_category, test_case_description, and simulation_configuration, matching UserConversationSimulationTestCase. Omit when the dataset uses these names or when using inline_user_conversation_simulation.
evaluation_level
enum | string
The level at which evaluation is performed.
enum
humanEvaluationPreview
HumanEvaluationPreviewRuleAction
Evaluation rule action for human evaluation.
type
enum
required
templateId
string
required
Identifier of a saved asset.
filter
EvaluationRuleFilter
Evaluation filter model.
agentName
string
required
Filter by agent name.
eventType
string | enum
required
Type of the evaluation rule event.
enum
schedule
Cron | Recurrence | OneTime
Base model for Trigger of the schedule.
Cron
CronTrigger
Cron based trigger.
type
enum
required
expression
string
required
Five-field cron expression that defines the schedule frequency: minute hour day-of-month month day-of-week.
timeZone
string
Time zone for the cron schedule. Defaults to UTC.
startTime
string
Start time for the cron schedule in ISO 8601 format.
endTime
string
End time for the cron schedule in ISO 8601 format.
Recurrence
RecurrenceTrigger
Recurrence based trigger.
type
enum
required
Type of the trigger.
startTime
string
Start time for the recurrence schedule in ISO 8601 format.
endTime
string
End time for the recurrence schedule in ISO 8601 format.
timeZone
string
Time zone for the recurrence schedule. Defaults to UTC.
interval
integer
required
Interval for the recurrence schedule.
schedule
Hourly | Daily | Weekly | Monthly
required
Recurrence schedule model.
Hourly
HourlyRecurrenceSchedule
Hourly recurrence schedule.
type
enum
required
Daily
DailyRecurrenceSchedule
Daily recurrence schedule.
type
enum
required
Daily recurrence type.
hours
integer[]
Hours for the recurrence schedule. If omitted, the service uses the current UTC hour when the schedule is created.
Weekly
WeeklyRecurrenceSchedule
Weekly recurrence schedule.
type
enum
required
Weekly recurrence type.
daysOfWeek
DayOfWeek[]
required
Days of the week for the recurrence schedule.
Monthly
MonthlyRecurrenceSchedule
Monthly recurrence schedule.
type
enum
required
Monthly recurrence type.
daysOfMonth
integer[]
required
Days of the month for the recurrence schedule.
OneTime
OneTimeTrigger
One-time trigger.
type
enum
required
triggerAt
string
required
Date and time for the one-time trigger in ISO 8601 format.
timeZone
string
Time zone for the one-time trigger. Defaults to UTC.
provisioningStatus
string | enum
Schedule provisioning status.
enum
error
Azure.Core.Foundations.Error
The error object.
code
string
required
One of a server-defined set of error codes.
message
string
required
A human-readable representation of the error.
target
string
The target of the error.
details
Azure.Core.Foundations.Error[]
An array of details about specific errors that led to this reported error.
innererror
Azure.Core.Foundations.InnerError
An object containing more specific information about the error. As per Azure REST API guidelines - https://aka.ms/AzureRestApiGuidelines#handling-errors.
code
string
One of a server-defined set of error codes.
enabled
boolean
required
Indicates whether the evaluation rule is enabled. Default is true.
systemData
object
required
System metadata for the evaluation rule.
Responses
The request has succeeded.
Body
id
string
required
Unique identifier for the evaluation rule.
displayName
string
Display Name for the evaluation rule.
description
string
Description for the evaluation rule.
action
continuousEvaluation | traceEvaluation | datasetEvaluation | humanEvaluationPreview
required
Evaluation action model.
continuousEvaluation
ContinuousEvaluationRuleAction
Evaluation rule action for continuous evaluation.
type
enum
required
evalId
string
required
Eval Id to add continuous evaluation runs to.
maxHourlyRuns
integer
Maximum number of evaluation runs allowed per hour.
samplingRate
number
Percentage (0-100] chance that a matching event triggers an evaluation. When omitted, the service-default is to evaluate every event, which is equivalent to setting a sampling rate of 100.
traceEvaluation
TraceEvaluationRuleAction
Configures a scheduled evaluation of observed agent traces.
type
enum
required
evalId
string
required
Identifier of the evaluation used to score the selected traces.
source
TracesEvalRunDataSource | TraceSourceEvalRunDataSource | TracesPreviewEvalRunDataSource | TraceSourcePreviewEvalRunDataSource
required
Existing trace-evaluation inputs supported by scheduled rules.
type
enum
required
The type of data source, always azure_ai_traces.
trace_ids
string[]
Collection of Agent trace identifiers that should be evaluated.
conversation_ids
string[]
Collection of conversation identifiers to filter traces by conversation.
agent_id
string
The agent ID used to filter traces for evaluation.
agent_name
string
The agent name used to filter traces for evaluation.
lookback_hours
integer
Lookback window (in hours) applied when retrieving traces from Application Insights. For scheduled evaluations this is inferred from the recurrence interval.
max_traces
integer
Sampling limit applied to traces retrieved for evaluation.
ingestion_delay_seconds
integer
The delay to apply for ingestion when querying traces.
type
enum
required
The type of data source, always azure_ai_trace_data_source.
trace_source
trace_id_source | conversation_id_source | agent_filter
Base class for polymorphic trace sources nested within a TraceSourceEvalRunDataSource.
trace_id_source
TraceIdTraceSource
A trace source that selects traces by explicit trace IDs.
type
enum
required
The type of trace source, always trace_id_source.
trace_ids
string[]
required
Collection of trace identifiers to evaluate.
conversation_id_source
ConversationIdTraceSource
A trace source that selects traces by conversation IDs.
type
enum
required
The type of trace source, always conversation_id_source.
conversation_ids
string[]
required
Collection of conversation identifiers to filter traces by.
agent_filter
AgentFilterTraceSource
A trace source that selects traces by agent reference with time-based filtering.
type
enum
required
The type of trace source, always agent_filter.
agent_id
string
The identifier of the agent to filter traces by.
agent_name
string
The name of the agent to filter traces by.
agent_version
string
The version of the agent to filter traces by.
max_traces
integer
Maximum number of traces to sample. Must be greater than 0.
filter_strategy
string | enum
Specifies the type of filter strategy used to select traces for evaluation.
enum
type
enum
required
The type of data source, always azure_ai_traces_preview.
trace_ids
string[]
Collection of Agent trace identifiers that should be evaluated.
conversation_ids
string[]
Collection of conversation identifiers to filter traces by conversation.
agent_id
string
The agent ID used to filter traces for evaluation.
agent_name
string
The agent name used to filter traces for evaluation.
lookback_hours
integer
Lookback window (in hours) applied when retrieving traces from Application Insights. For scheduled evaluations this is inferred from the recurrence interval.
max_traces
integer
Sampling limit applied to traces retrieved for evaluation.
ingestion_delay_seconds
integer
The delay to apply for ingestion when querying traces.
type
enum
required
The type of data source, always azure_ai_trace_data_source_preview.
trace_source
trace_id_source | conversation_id_source | agent_filter
Base class for polymorphic trace sources nested within a TraceSourceEvalRunDataSource.
trace_id_source
TraceIdTraceSource
A trace source that selects traces by explicit trace IDs.
type
enum
required
The type of trace source, always trace_id_source.
trace_ids
string[]
required
Collection of trace identifiers to evaluate.
conversation_id_source
ConversationIdTraceSource
A trace source that selects traces by conversation IDs.
type
enum
required
The type of trace source, always conversation_id_source.
conversation_ids
string[]
required
Collection of conversation identifiers to filter traces by.
agent_filter
AgentFilterTraceSource
A trace source that selects traces by agent reference with time-based filtering.
type
enum
required
The type of trace source, always agent_filter.
agent_id
string
The identifier of the agent to filter traces by.
agent_name
string
The name of the agent to filter traces by.
agent_version
string
The version of the agent to filter traces by.
max_traces
integer
Maximum number of traces to sample. Must be greater than 0.
filter_strategy
string | enum
Specifies the type of filter strategy used to select traces for evaluation.
enum
evaluation_level
enum | string
The level at which evaluation is performed.
enum
datasetEvaluation
DatasetEvaluationRuleAction
Configures a scheduled evaluation of dataset inputs.
type
enum
required
evalId
string
required
Identifier of the evaluation used to score the dataset.
source
OpenAI.CreateEvalJsonlRunDataSource | CsvEvalRunDataSource | OpenAI.CreateEvalCompletionsRunDataSource | TargetCompletionEvalRunDataSource | UserConversationSimulationEvalRunDataSource | UserConversationSimulationPreviewEvalRunDataSource
required
Dataset inputs supported by scheduled rules. Targets are configured within the selected source.
type
enum
required
The type of data source. Always jsonl.
source
OpenAI.EvalJsonlFileContentSource | OpenAI.EvalJsonlFileIdSource
required
Determines what populates the item namespace in the data source.
type
enum
required
The type of jsonl source. Always file_content.
content
OpenAI.EvalJsonlFileContentSourceContent[]
required
The content of the jsonl file.
type
enum
required
The type of jsonl source. Always file_id.
id
string
required
The identifier of the file.
type
enum
required
The type of data source, always csv.
source
EvalCsvFileIdSource
required
Represents a reference to an uploaded CSV file used as a source for evaluation data.
type
enum
required
The type of source, always file_id.
id
string
required
The identifier of the uploaded CSV file.
type
enum
required
The type of run data source. Always completions.
input_messages
OpenAI.CreateEvalCompletionsRunDataSourceInputMessagesTemplate | OpenAI.CreateEvalCompletionsRunDataSourceInputMessagesItemReference
Used when sampling from a model. Dictates the structure of the messages passed into the model. Can either be a reference to a prebuilt trajectory (ie, item.input_trajectory), or a template with variable references to the item namespace.
type
enum
required
type
enum
required
sampling_params
OpenAI.CreateEvalCompletionsRunDataSourceSamplingParams
reasoning_effort
string | enum | null
Constrains effort on reasoning for reasoning models. Currently supported values are none, minimal, low, medium, high, xhigh, and max. Reducing reasoning effort can result in faster responses and fewer tokens used on reasoning in a response. Not all reasoning models support every value. See the reasoning guide for model-specific support.
enum
response_format
OpenAI.ResponseFormatText | OpenAI.ResponseFormatJsonSchema | OpenAI.ResponseFormatJsonObject
type
enum
required
The type of response format being defined. Always text.
type
enum
required
The type of response format being defined. Always json_schema.
json_schema
OpenAI.ResponseFormatJsonSchemaJsonSchema
required
schema
OpenAI.ResponseFormatJsonSchemaSchema
The schema for the response format, described as a JSON Schema object. Learn how to build JSON schemas here.
type
enum
required
The type of response format being defined. Always json_object.
model
string
The name of the model to use for generating completions (e.g. "o3-mini").
source
OpenAI.EvalJsonlFileContentSource | OpenAI.EvalJsonlFileIdSource | OpenAI.EvalStoredCompletionsSource
required
Determines what populates the item namespace in this run's data source.
type
enum
required
The type of jsonl source. Always file_content.
content
OpenAI.EvalJsonlFileContentSourceContent[]
required
The content of the jsonl file.
type
enum
required
The type of jsonl source. Always file_id.
id
string
required
The identifier of the file.
type
enum
required
The type of source. Always stored_completions.
metadata
OpenAI.Metadata | null
Set of 16 key-value pairs that can be attached to an object. This can be useful for storing additional information about the object in a structured format, and querying for objects via API or the dashboard. Keys are strings with a maximum length of 64 characters. Values are strings with a maximum length of 512 characters.
type
enum
required
The type of data source, always azure_ai_target_completions.
input_messages
OpenAI.CreateEvalCompletionsRunDataSourceInputMessagesItemReference
type
enum
required
source
OpenAI.EvalJsonlFileContentSource | OpenAI.EvalJsonlFileIdSource
required
The source configuration for inline or file data.
type
enum
required
The type of jsonl source. Always file_content.
content
OpenAI.EvalJsonlFileContentSourceContent[]
required
The content of the jsonl file.
type
enum
required
The type of jsonl source. Always file_id.
id
string
required
The identifier of the file.
target
azure_ai_model | azure_ai_agent
required
Base class for targets with discriminator support.
azure_ai_model
AzureAIModelTarget
A target that identifies an Azure AI model. See evaluate administrator-connected models.
type
enum
required
The type of target, always azure_ai_model.
model
string
The name of the model deployment to invoke as the evaluation target.
sampling_params
ModelSamplingParams
Represents a set of parameters used to control the sampling behavior of a language model during text generation.
temperature
number
The temperature parameter for sampling. Defaults to 1.0.
top_p
number
The top-p parameter for nucleus sampling. Defaults to 1.0.
seed
integer
The random seed for reproducibility. Defaults to 42.
max_completion_tokens
integer
The maximum number of tokens allowed in the completion.
azure_ai_agent
AzureAIAgentTarget
A target that identifies an Azure AI agent. See evaluate model or agent targets.
type
enum
required
The type of target, always azure_ai_agent.
name
string
required
The name of the Azure AI agent to invoke as the evaluation target.
version
string
The version of the Azure AI agent. When omitted, the latest version is used.
tool_descriptions
ToolDescription[]
Optional descriptions of tools available to the agent.
tools
OpenAI.Tool[]
Tool definitions made available when invoking the agent.
item_generation_params
red_team | red_team_seed_prompts | red_team_taxonomy | response_retrieval | target_completion | synthetic_data_gen
Configures how the service prepares input items for an evaluation run before applying testing criteria. Depending on type, preparation can generate new inputs, retrieve existing responses, transform source data, or simulate conversations. Each type selects a workflow-specific model with the parameters required by that preparation workflow. These parameters do not define the evaluated target or the testing criteria.
red_team
RedTeamItemGenerationParams
Represents the parameters for red team item generation.
type
enum
required
The type of item generation parameters.
attack_strategies
AttackStrategy[]
required
The collection of attack strategies to be used.
num_turns
integer
required
The number of turns allowed in the game.
red_team_seed_prompts
RedTeamSeedPromptsItemGenerationParams
Represents the parameters for red team seed prompts item generation.
type
enum
required
The type of item generation parameters, always red_team_seed_prompts.
attack_strategies
AttackStrategy[]
required
The collection of attack strategies to be used.
num_turns
integer
required
The number of turns allowed in the game.
source
OpenAI.EvalJsonlFileContentSource
required
type
enum
required
The type of jsonl source. Always file_content.
content
OpenAI.EvalJsonlFileContentSourceContent[]
required
The content of the jsonl file.
red_team_taxonomy
RedTeamTaxonomyItemGenerationParams
Represents the parameters for red team taxonomy item generation.
type
enum
required
The type of item generation parameters, always red_team_taxonomy.
attack_strategies
AttackStrategy[]
required
The collection of attack strategies to be used.
num_turns
integer
required
The number of turns allowed in the game.
source
OpenAI.EvalJsonlFileContentSource
required
type
enum
required
The type of jsonl source. Always file_content.
content
OpenAI.EvalJsonlFileContentSourceContent[]
required
The content of the jsonl file.
response_retrieval
ResponseRetrievalItemGenerationParams
Parameters for retrieving deployed conversations for evaluation. See evaluate deployed conversations.
type
enum
required
The type of item generation parameters, always response_retrieval.
max_num_turns
integer
required
The maximum number of turns of chat history to evaluate.
data_mapping
object
required
Maps response_id to the source field that contains a stored Responses API response identifier. For example, {"response_id": "{{item.resp_id}}"}. Field names are case-sensitive.
source
OpenAI.EvalJsonlFileContentSource | OpenAI.EvalJsonlFileIdSource
required
The inline JSONL content containing the response identifiers to retrieve. The azure_ai_responses data source supports file_content only; providing a file_id results in a 400 Bad Request response.
type
enum
required
The type of jsonl source. Always file_content.
content
OpenAI.EvalJsonlFileContentSourceContent[]
required
The content of the jsonl file.
type
enum
required
The type of jsonl source. Always file_id.
id
string
required
The identifier of the file.
target_completion
TargetCompletionItemGenerationParams
Parameters for generating target completions for simulated conversations. See simulate conversations.
type
enum
required
The type of item generation parameters, always target_completion.
max_concurrency
integer
The maximum number of target completions to execute concurrently. The services defaults to 1 if a value is not specified by the caller, which invokes the target sequentially.
synthetic_data_gen
SyntheticDataGenerationItemGenerationParams
Represents the parameters for synthetic data generation item generation.
type
enum
required
The type of item generation parameters.
samples_count
integer
required
The maximum number of data samples to generate.
prompt
string
The prompt used to generate synthetic data. This property is optional when the target is an agent that already has instructions.
model_deployment_name
string
required
The name of the model deployment to use for generating synthetic data.
output_dataset_name
string
The name of the output dataset where generated synthetic data will be stored. If not provided, service generates dataset name automatically.
output_dataset_id
string
The identifier of the output dataset where generated synthetic data is stored. The generated data is a jsonl file with columns id, query and test_description.
sources
OpenAI.EvalJsonlFileIdSource[]
required
The optional seed data content source files for data generation.
type
enum
required
The type of data source, always azure_ai_user_conversation_simulation.
source
OpenAI.EvalJsonlFileContentSource | OpenAI.EvalJsonlFileIdSource | UserConversationSimulationInlineTestCasesSource
required
Sources supported by user conversation simulation.
type
enum
required
The type of jsonl source. Always file_content.
content
OpenAI.EvalJsonlFileContentSourceContent[]
required
The content of the jsonl file.
type
enum
required
The type of jsonl source. Always file_id.
id
string
required
The identifier of the file.
type
enum
required
The source type. Always inline_user_conversation_simulation.
test_cases
UserConversationSimulationTestCase[]
required
Test scenarios from which user conversations are simulated. At least one test case is required.
target
azure_ai_model | azure_ai_agent
required
Base class for targets with discriminator support.
azure_ai_model
AzureAIModelTarget
A target that identifies an Azure AI model. See evaluate administrator-connected models.
type
enum
required
The type of target, always azure_ai_model.
model
string
The name of the model deployment to invoke as the evaluation target.
sampling_params
ModelSamplingParams
Represents a set of parameters used to control the sampling behavior of a language model during text generation.
temperature
number
The temperature parameter for sampling. Defaults to 1.0.
top_p
number
The top-p parameter for nucleus sampling. Defaults to 1.0.
seed
integer
The random seed for reproducibility. Defaults to 42.
max_completion_tokens
integer
The maximum number of tokens allowed in the completion.
azure_ai_agent
AzureAIAgentTarget
A target that identifies an Azure AI agent. See evaluate model or agent targets.
type
enum
required
The type of target, always azure_ai_agent.
name
string
required
The name of the Azure AI agent to invoke as the evaluation target.
version
string
The version of the Azure AI agent. When omitted, the latest version is used.
tool_descriptions
ToolDescription[]
Optional descriptions of tools available to the agent.
tools
OpenAI.Tool[]
Tool definitions made available when invoking the agent.
model_configuration
EvaluationModelConfiguration
required
Configures the model that plays the simulated user. This model is separate from the evaluated target.
model
string
required
Model deployment that generates simulated user turns, in the format {connectionName}/modelDeploymentName.
sampling_params
ModelSamplingParams
Represents a set of parameters used to control the sampling behavior of a language model during text generation.
temperature
number
The temperature parameter for sampling. Defaults to 1.0.
top_p
number
The top-p parameter for nucleus sampling. Defaults to 1.0.
seed
integer
The random seed for reproducibility. Defaults to 42.
max_completion_tokens
integer
The maximum number of tokens allowed in the completion.
voice_model
azure-standard
Voice configuration used to convert text to speech for evaluation through the Voice Live endpoint.
azure-standard
EvaluationAzureStandardVoiceModelConfiguration
Configures an Azure standard neural voice used to convert text to speech for evaluation through the Voice Live endpoint.
type
enum
required
The voice kind. Always azure-standard.
name
string
required
The Azure neural voice name.
temperature
number
The synthesis temperature, from 0 to 1. When omitted, the service defaults to the underlying voice model's default temperature.
default_simulation_configuration
UserConversationSimulationDefaultConfiguration
Configures default conversation behavior and optional run-wide persistence of generated conversations.
max_num_turns
integer
Hard limit on turns in each conversation. When omitted, the service defaults to 20.
conversation_repetitions
integer
Number of independent conversation repetitions for each test case. Defaults to 1 when not specified at either the data-source or test-case level.
desired_num_turns
integer
Target number of turns in each conversation. The effective value cannot exceed the effective max_num_turns. When omitted, no target is set and the simulation model determines the conversation length dynamically from the scenario.
audio_effects
UserConversationSimulationAudioEffectsConfiguration
Configures effects applied to simulated conversation audio.
effects
UserConversationSimulationAudioEffect[]
Effects to apply to simulated audio.
volume_percentage
integer
Volume of the configured audio effects, as a percentage from 1 through 100. When omitted, the service defaults to 15.
user_behavior
UserConversationSimulationUserBehaviorConfiguration
Configures conversation behavior for the simulated user.
interruption
default
Configuration for simulated user interruption behavior.
default
UserConversationSimulationDefaultInterruptionConfiguration
Configures the default interruption behavior for the simulated user.
type
enum
required
The interruption type, always default.
enable_conversation_dataset_generation
boolean
Whether to write all generated conversations to a versioned Foundry dataset. When enabled, the completed run includes a simulated_user_conversations entry in output_datasets. When omitted, the service defaults to false.
output_conversation_dataset_name
string
Name of the dataset to which generated conversations are written. This property is used only when enable_conversation_dataset_generation is true. When omitted, the service generates a dataset name.
data_mapping
object
Maps JSONL dataset attributes to user conversation simulation inputs when their names differ from the defaults. The default attribute names are test_case_id, test_case_category, test_case_description, and simulation_configuration, matching UserConversationSimulationTestCase. Omit when the dataset uses these names or when using inline_user_conversation_simulation.
type
enum
required
The type of data source, always azure_ai_user_conversation_simulation_preview.
source
OpenAI.EvalJsonlFileContentSource | OpenAI.EvalJsonlFileIdSource | UserConversationSimulationInlineTestCasesSource
required
Sources supported by user conversation simulation.
type
enum
required
The type of jsonl source. Always file_content.
content
OpenAI.EvalJsonlFileContentSourceContent[]
required
The content of the jsonl file.
type
enum
required
The type of jsonl source. Always file_id.
id
string
required
The identifier of the file.
type
enum
required
The source type. Always inline_user_conversation_simulation.
test_cases
UserConversationSimulationTestCase[]
required
Test scenarios from which user conversations are simulated. At least one test case is required.
target
azure_ai_model | azure_ai_agent
required
Base class for targets with discriminator support.
azure_ai_model
AzureAIModelTarget
A target that identifies an Azure AI model. See evaluate administrator-connected models.
type
enum
required
The type of target, always azure_ai_model.
model
string
The name of the model deployment to invoke as the evaluation target.
sampling_params
ModelSamplingParams
Represents a set of parameters used to control the sampling behavior of a language model during text generation.
temperature
number
The temperature parameter for sampling. Defaults to 1.0.
top_p
number
The top-p parameter for nucleus sampling. Defaults to 1.0.
seed
integer
The random seed for reproducibility. Defaults to 42.
max_completion_tokens
integer
The maximum number of tokens allowed in the completion.
azure_ai_agent
AzureAIAgentTarget
A target that identifies an Azure AI agent. See evaluate model or agent targets.
type
enum
required
The type of target, always azure_ai_agent.
name
string
required
The name of the Azure AI agent to invoke as the evaluation target.
version
string
The version of the Azure AI agent. When omitted, the latest version is used.
tool_descriptions
ToolDescription[]
Optional descriptions of tools available to the agent.
tools
OpenAI.Tool[]
Tool definitions made available when invoking the agent.
model_configuration
EvaluationModelConfiguration
required
Configures the model that plays the simulated user. This model is separate from the evaluated target.
model
string
required
Model deployment that generates simulated user turns, in the format {connectionName}/modelDeploymentName.
sampling_params
ModelSamplingParams
Represents a set of parameters used to control the sampling behavior of a language model during text generation.
temperature
number
The temperature parameter for sampling. Defaults to 1.0.
top_p
number
The top-p parameter for nucleus sampling. Defaults to 1.0.
seed
integer
The random seed for reproducibility. Defaults to 42.
max_completion_tokens
integer
The maximum number of tokens allowed in the completion.
voice_model
azure-standard
Voice configuration used to convert text to speech for evaluation through the Voice Live endpoint.
azure-standard
EvaluationAzureStandardVoiceModelConfiguration
Configures an Azure standard neural voice used to convert text to speech for evaluation through the Voice Live endpoint.
type
enum
required
The voice kind. Always azure-standard.
name
string
required
The Azure neural voice name.
temperature
number
The synthesis temperature, from 0 to 1. When omitted, the service defaults to the underlying voice model's default temperature.
default_simulation_configuration
UserConversationSimulationDefaultConfiguration
Configures default conversation behavior and optional run-wide persistence of generated conversations.
max_num_turns
integer
Hard limit on turns in each conversation. When omitted, the service defaults to 20.
conversation_repetitions
integer
Number of independent conversation repetitions for each test case. Defaults to 1 when not specified at either the data-source or test-case level.
desired_num_turns
integer
Target number of turns in each conversation. The effective value cannot exceed the effective max_num_turns. When omitted, no target is set and the simulation model determines the conversation length dynamically from the scenario.
audio_effects
UserConversationSimulationAudioEffectsConfiguration
Configures effects applied to simulated conversation audio.
effects
UserConversationSimulationAudioEffect[]
Effects to apply to simulated audio.
volume_percentage
integer
Volume of the configured audio effects, as a percentage from 1 through 100. When omitted, the service defaults to 15.
user_behavior
UserConversationSimulationUserBehaviorConfiguration
Configures conversation behavior for the simulated user.
interruption
default
Configuration for simulated user interruption behavior.
default
UserConversationSimulationDefaultInterruptionConfiguration
Configures the default interruption behavior for the simulated user.
type
enum
required
The interruption type, always default.
enable_conversation_dataset_generation
boolean
Whether to write all generated conversations to a versioned Foundry dataset. When enabled, the completed run includes a simulated_user_conversations entry in output_datasets. When omitted, the service defaults to false.
output_conversation_dataset_name
string
Name of the dataset to which generated conversations are written. This property is used only when enable_conversation_dataset_generation is true. When omitted, the service generates a dataset name.
data_mapping
object
Maps JSONL dataset attributes to user conversation simulation inputs when their names differ from the defaults. The default attribute names are test_case_id, test_case_category, test_case_description, and simulation_configuration, matching UserConversationSimulationTestCase. Omit when the dataset uses these names or when using inline_user_conversation_simulation.
evaluation_level
enum | string
The level at which evaluation is performed.
enum
humanEvaluationPreview
HumanEvaluationPreviewRuleAction
Evaluation rule action for human evaluation.
type
enum
required
templateId
string
required
Identifier of a saved asset.
filter
EvaluationRuleFilter
Evaluation filter model.
agentName
string
required
Filter by agent name.
eventType
string | enum
required
Type of the evaluation rule event.
enum
schedule
Cron | Recurrence | OneTime
Base model for Trigger of the schedule.
Cron
CronTrigger
Cron based trigger.
type
enum
required
expression
string
required
Five-field cron expression that defines the schedule frequency: minute hour day-of-month month day-of-week.
timeZone
string
Time zone for the cron schedule. Defaults to UTC.
startTime
string
Start time for the cron schedule in ISO 8601 format.
endTime
string
End time for the cron schedule in ISO 8601 format.
Recurrence
RecurrenceTrigger
Recurrence based trigger.
type
enum
required
Type of the trigger.
startTime
string
Start time for the recurrence schedule in ISO 8601 format.
endTime
string
End time for the recurrence schedule in ISO 8601 format.
timeZone
string
Time zone for the recurrence schedule. Defaults to UTC.
interval
integer
required
Interval for the recurrence schedule.
schedule
Hourly | Daily | Weekly | Monthly
required
Recurrence schedule model.
Hourly
HourlyRecurrenceSchedule
Hourly recurrence schedule.
type
enum
required
Daily
DailyRecurrenceSchedule
Daily recurrence schedule.
type
enum
required
Daily recurrence type.
hours
integer[]
Hours for the recurrence schedule. If omitted, the service uses the current UTC hour when the schedule is created.
Weekly
WeeklyRecurrenceSchedule
Weekly recurrence schedule.
type
enum
required
Weekly recurrence type.
daysOfWeek
DayOfWeek[]
required
Days of the week for the recurrence schedule.
Monthly
MonthlyRecurrenceSchedule
Monthly recurrence schedule.
type
enum
required
Monthly recurrence type.
daysOfMonth
integer[]
required
Days of the month for the recurrence schedule.
OneTime
OneTimeTrigger
One-time trigger.
type
enum
required
triggerAt
string
required
Date and time for the one-time trigger in ISO 8601 format.
timeZone
string
Time zone for the one-time trigger. Defaults to UTC.
provisioningStatus
string | enum
Schedule provisioning status.
enum
error
Azure.Core.Foundations.Error
The error object.
code
string
required
One of a server-defined set of error codes.
message
string
required
A human-readable representation of the error.
target
string
The target of the error.
details
Azure.Core.Foundations.Error[]
An array of details about specific errors that led to this reported error.
innererror
Azure.Core.Foundations.InnerError
An object containing more specific information about the error. As per Azure REST API guidelines - https://aka.ms/AzureRestApiGuidelines#handling-errors.
code
string
One of a server-defined set of error codes.
enabled
boolean
required
Indicates whether the evaluation rule is enabled. Default is true.
systemData
object
required
System metadata for the evaluation rule.
Request
PUT /evaluationrules/{id} HTTP/1.1Response
Status: 200
Content-Type: application/jsonStatus: 201
Content-Type: application/jsonStatus: 4XX
Content-Type: application/jsonStatus: 5XX
Content-Type: application/json