Gating a Merge on an Eval Score in Azure Pipelines

If your Azure Pipelines eval gate runs on pushes to main but never on a pull request, the YAML is not the problem. Microsoft’s documentation is explicit: for an Azure Repos Git repository you cannot configure a PR trigger in the YAML file, and the functionality is implemented by a branch policy instead. Why your pr trigger does nothing The pr: key exists in the Azure Pipelines YAML schema, and it works — for GitHub and Bitbucket Cloud repositories. For Azure Repos Git it is inert. The Azure Repos Git documentation states that pull request triggers are implemented using branch policies, and that to enable PR validation you configure the Build validation policy on the target branch. A pr: block in the file is not an error and produces no warning; it simply never causes a run. Two related things surprise people once the policy exists. Draft pull requests do not trigger a pipeline even with a branch policy configured, so a gate that seems not to run may be running against a draft. And you must be a project administrator of the project to configure validation builds at all, which is why this is often the step that a developer cannot complete themselves. This is a product behaviour rather than a version detail, but it is the kind of thing that changes. Check the Azure Repos Git page in Microsoft’s Azure Pipelines documentation before assuming it still holds. The pipeline A single-stage pipeline is enough. The CI trigger below covers pushes; the pull request path comes from the policy in the next section, and no pr: key appears at all because on Azure Repos it would only be misleading to a reader. trigger : branches : include : - main paths : exclude : - docs/* pool : vmImage : ubuntu-latest variables : - group : llm-eval-keys - name : EVAL_MODEL value : gpt-4.1-mini-2025-04-14 steps : - task : UsePythonVersion@0 inputs : versionSpec : ' 3.12' - script : pip install -r evals/requirements.txt displayName : Install eval dependencies - script : | python -m evals.run \ --cases