How to run an evaluation in GitHub Action (preview)

This GitHub Action enables offline evaluation of Microsoft Foundry Agents within your CI/CD pipelines. It streamlines the offline evaluation process, so you can identify potential problems and make improvements before releasing an update to production.

To use this action, provide a data set with test queries and a list of evaluators. This action invokes your agents with the queries, runs the evaluations, and generates a summary report.

Features

• Agent Evaluation: Automate pre-production assessment of Microsoft Foundry agents in your CI/CD workflow.
• Evaluators: Use any evaluators from the Foundry evaluator catalog.
• Statistical Analysis: Evaluation results include confidence intervals and test for statistical significance to determine if changes are meaningful and not due to random variation.

Evaluator categories

• Agent evaluators: Process and system-level evaluators for agent workflows.
• RAG evaluators: Evaluate end-to-end and retrieval processes in RAG systems.
• Risk and safety evaluators: Assess risks and safety concerns in responses.
• General purpose evaluators: Quality evaluation such as coherence and fluency.
• OpenAI-based graders: Use OpenAI graders including string check, text similarity, score/label model.
• Custom evaluators: Define your own custom evaluators using Python code or LLM-as-a-judge patterns.

Prerequisites

• A project. To learn more, see Create a project.
• A Foundry agent.

How to set up AI agent evaluations

AI agent evaluations input

Parameters

Data file

The input data file should be a JSON file with the following structure:

Basic sample data file

{
  "name": "test-data",
  "evaluators": [
    "builtin.fluency",
    "builtin.task_adherence",
    "builtin.violence"
  ],
  "data": [
    {
      "query": "Tell me about Tokyo disneyland"
    },
    {
      "query": "How do I install Python?"
    }
  ]
}

Additional sample data files

AI agent evaluations workflow

To use the GitHub Action, add the GitHub Action to your CI/CD workflows. Specify the trigger criteria, such as on commit, and the file paths to trigger your automated workflows.

This example shows how you can run AI Agent Evaluation when you compare different agents by using agent IDs.

name: "AI Agent Evaluation"

on:
  workflow_dispatch:
  push:
    branches:
      - main

permissions:
  id-token: write
  contents: read

jobs:
  run-action:
    runs-on: ubuntu-latest
    steps:
      - name: Checkout
        uses: actions/checkout@v4

      - name: Azure login using Federated Credentials
        uses: azure/login@v2
        with:
          client-id: ${{ vars.AZURE_CLIENT_ID }}
          tenant-id: ${{ vars.AZURE_TENANT_ID }}
          subscription-id: ${{ vars.AZURE_SUBSCRIPTION_ID }}

      - name: Run Evaluation
        uses: microsoft/ai-agent-evals@v3-beta
        with:
          # Replace placeholders with values for your Foundry Project
          azure-ai-project-endpoint: "<your-ai-project-endpoint>"
          deployment-name: "<your-deployment-name>"
          agent-ids: "<your-ai-agent-ids>"
          data-path: ${{ github.workspace }}/path/to/your/data-file

AI agent evaluations output

Evaluation results are output to the summary section for each AI Evaluation GitHub Action run under Actions in GitHub. The report shows evaluation scores for each metric, confidence intervals, and — when you evaluate multiple agents — a pairwise statistical comparison that indicates whether differences are meaningful or within random variation.

The following screenshot shows a sample report comparing two agents.

Use the azd evaluation extension as a quality gate

You can use the Azure Developer CLI (azd) evaluation extension to run a registered evaluation and block a pull request when its pass rate is below a threshold. This workflow uses GitHub OpenID Connect (OIDC), so it doesn't store a client secret.

Prepare the repository

Before you configure the workflow, ensure you have:

• An Azure subscription and a Foundry project that contains the evaluation target.
• Admin access to configure GitHub Actions variables, secrets, environments, and protection rules.
• A checked-in evaluation definition and deterministic evaluation dataset.
• The Azure Developer CLI (azd), version 1.33.0 or later, which the evaluation extension requires. For installation instructions, see Install the Azure Developer CLI.
• The azure.ai.evaluations extension, which provides the evaluation commands. The Foundry azd package (microsoft.foundry) doesn't include it.

Install the azure.ai.evaluations extension and verify that the evaluation commands are available. In your workflow, run the same commands in a step after you install azd. For more information about the Foundry extensions, see Install the Azure Developer CLI Foundry extensions.

azd ext install azure.ai.evaluations
azd ai eval --help

Configure GitHub OIDC and least privilege

Create a Microsoft Entra application or user-assigned managed identity, and add a federated identity credential for the token issued to the workflow. Use these fixed values:

A repository in an organization or enterprise that uses an immutable custom subject can have a subject with this shape:

repository_owner_id:<owner-id>:repository_id:<repo-id>:pull_request

Reference: OpenID Connect reference

Don't copy owner or repository IDs from another repository. Request a token in your workflow, decode its payload without printing the token, and inspect only the iss, aud, and sub claims. Configure the federated credential to match the actual claims exactly. GitHub's standard subject for a job that uses an environment can have a different shape, and an organization or enterprise can customize the subject claim.

For Microsoft Entra workload identity setup, see Authenticate to Azure from GitHub Actions by OpenID Connect.

Assign the identity only these roles:

• Foundry User at the Foundry project scope.
• Contributor at the dedicated resource group scope, if you reconcile with azd up.

The resource group role is required when you reconcile with azd up, because azd up records an Azure Resource Manager deployment when it provisions. Don't assign Contributor at the subscription scope.

Protect and configure the live environment

Create a protected GitHub environment named live-evaluation. Add required reviewers and restrict deployment branches as appropriate for your repository. For more information, see Managing environments for deployment.

Store the workflow values as follows. Don't add AZURE_CLIENT_SECRET; OIDC replaces it.

Keep AZD_CONFIG_DIR under RUNNER_TEMP, and authenticate noninteractively:

azd auth login `
  --client-id $env:AZURE_CLIENT_ID `
  --tenant-id $env:AZURE_TENANT_ID `
  --federated-credential-provider github `
  --no-prompt

Reference: Azure Developer CLI reference

The job must have contents: read and id-token: write permissions. Gate it so that only same-repository pull requests or a manual dispatch from the default branch can enter the protected environment. Fork pull requests must skip the job, and the workflow must not use pull_request_target.

on:
  pull_request:
  workflow_dispatch:

permissions:
  contents: read
  id-token: write

jobs:
  live-evaluation:
    if: >-
      vars.AZD_EVAL_LIVE_PR_GATE == 'true' &&
      ((github.event_name == 'pull_request' &&
      github.event.pull_request.head.repo.full_name == github.repository) ||
      (github.event_name == 'workflow_dispatch' &&
      github.ref_name == github.event.repository.default_branch))
    environment:
      name: live-evaluation

Reference: GitHub Actions contexts reference

Pin each GitHub Action to a reviewed full commit SHA.

Provide an inert infrastructure template when azd up requires one

If you reconcile an evaluation-only project with azd up instead of azd ai eval create, the project doesn't provision workload resources. If azd up --no-prompt reports that it can't find or compile infra/main.bicep, add that file with this inert nested deployment:

targetScope = 'resourceGroup'

resource reconciliationMarker 'Microsoft.Resources/deployments@2022-09-01' = {
  name: 'azd-eval-hero-reconciliation'
  properties: {
    mode: 'Incremental'
    template: {
      '$schema': 'https://schema.management.azure.com/schemas/2019-04-01/deploymentTemplate.json#'
      contentVersion: '1.0.0.0'
      resources: []
    }
  }
}

The inner template contains no workload resources. The marker creates only an Azure Resource Manager deployment record.

For more information, see Microsoft.Resources deployments.

Run the exact evaluation

Keep the lifecycle in this order:

1. Reconcile the checked-in evaluation definition. Run azd up --no-prompt when the project provisions infrastructure. For an evaluation-only project, run azd ai eval create ci-f1-quality --no-prompt instead. This command registers the declared datasets, evaluators, and evaluation without provisioning infrastructure.
2. Run azd ai eval run start --eval ci-f1-quality --no-prompt --no-wait -o json.
3. Read .run_id from the captured JSON without writing the value to the log.
4. Run azd ai eval run show <run-id> --eval ci-f1-quality --wait --fail-on pass-rate=0.8 --no-prompt -o json.
5. Export that exact run with azd ai eval run output export <run-id> --eval ci-f1-quality --format json --output-file <runner-temp-file> --no-prompt.
6. Validate the completed status, aggregate counts, exported rows, and allowlisted evidence before upload.

Capture raw standard output and standard error in RUNNER_TEMP. Store the run ID in memory or a runner-temporary file, and use it only as the positional argument for the show and output export commands.

Use a separate GitHub Actions step for each lifecycle phase. On success, print only a sanitized command line and summary, such as reconciliation completed, run accepted with its ID redacted, final status and counts, gate outcome, exported row count, and the allowlisted evidence preview.

For --fail-on pass-rate=0.8, pass rate is passed test cases divided by total test cases. Failed, errored, skipped, and otherwise unpassed rows count against the run. A valid completed run below the threshold is a quality breach. The azd CLI collapses an extension's exit code, so the command exits with code 1 for a quality breach and for an operational failure. A quality breach prints the gate reason and the message evaluation quality gate not met. Authentication, network, malformed or incomplete results, and export inconsistencies are operational failures.

Upload only allowlisted evidence

Create a new evidence document from the validated result instead of uploading raw CLI or service output. Keep only aggregate status, counts, and threshold, plus these fields for each reviewed sample: query, response, ground_truth, score, and passed.

{
  "status": "completed",
  "counts": {
    "passed": 2,
    "failed": 0,
    "errored": 0
  },
  "threshold": 0.8,
  "samples": [
    {
      "query": "<allowlisted query>",
      "response": "<allowlisted response>",
      "ground_truth": "<allowlisted ground truth>",
      "score": 1.0,
      "passed": true
    }
  ]
}

Before upload, validate that the artifact directory contains only the expected evidence file. Never print or upload JWTs, endpoints, run IDs, credentials, userinfo, URL query strings, or raw authentication, CLI, or service payloads. Keep raw files in runner-temporary storage, redact captured failure diagnostics, and delete the files in an always() or finally cleanup path.

Verify the gate

Run the workflow manually from the default branch before you enable pull request gating. Confirm that:

1. OIDC authentication succeeds without a client secret.
2. The reconcile step registers the checked-in evaluation.
3. The asynchronous start is accepted without displaying the run ID.
4. The show command reports completed and enforces the 0.8 pass-rate threshold.
5. The export contains only rows from the exact run.
6. The uploaded artifact contains only the allowlisted evidence schema.
7. Raw runner-temporary files are deleted on success and failure.

Troubleshoot the gate

After verification, remove disposable test resources and role assignments that you no longer need. Delete temporary federated credentials, and keep the dedicated resource group only if the quality gate continues to use it.

Related content

• Evaluation in Azure DevOps
• Run agent evaluations with the azd CLI
• Install the Azure Developer CLI Foundry extensions
• How to evaluate generative AI models and applications with Microsoft Foundry
• How to view evaluation results in Foundry portal