Create Custom Scorers for Agent Testing

Custom scorers allow you to define evaluation logic for agent testing. Standard Expectations cover common test scenarios like topic matching and response coherence, but custom scorers create expectations tailored to your business requirements.

Custom scorers evaluate agent behavior at different levels of granularity. A custom scorer can test an entire conversation, a single interaction, or a specific moment within an interaction. Scorers use a prompt template to evaluate agent behavior automatically with an LLM.

Use the AiAgentScorerDefinition Metadata API type to define custom scorers and deploy them to your org.

How Custom Scorers Work

A custom scorer evaluates agent behavior and produces a result that maps to an outcome: pass, fail, or not applicable. Each scorer has two key components:

Engine: The evaluation logic. Use a PromptTemplate engine to assess the agent’s behavior with an LLM.
Output mapping: Rules that translate the engine’s result into a pass, fail, or not-applicable outcome.

In this guide, we define a custom scorer that uses a prompt template to detect whether a customer dropped off before the conversation resolved.

Prerequisites

Agentforce is enabled in your org with at least one active agent. See Set up Agents in Salesforce Help.
If your scorer uses the PromptTemplate engine type, the prompt template must exist in your org or get deployed alongside the scorer. See Deploy a Scorer with a Prompt Template for details on deploying both together.

AiAgentScorerDefinition Reference

Create an AiAgentScorerDefinition metadata component to define your scorer. The component exists in the aiAgentScorerDefinitions folder with the .aiAgentScorerDefinition file suffix.

inputScope

The scorer defines two inputScope fields that work together to control what data the scorer evaluates:

Field	Valid values	Description
Top-level `inputScope`	`Session`	The data the scorer’s evaluation logic (its prompt template or manual scores) operates on.
`agentAssociation.inputScope`	`Session`, `Intent`	The unit of agent activity that the scorer runs against. Use `Intent` to score each intent in a session individually.

To reference the latest interaction inside a session-scoped scorer, use the getLastInteraction invocable action within your prompt template.

Scorer Definition Reference

Field	Type	Description
`inputScope`	string	Required. The scope of agent data the scorer’s evaluation logic operates on. Valid value: `Session`.
`scorerType`	string	Required. The scorer’s output shape. Valid values: `Predefined`, `OpenEnded`. Use `Predefined` when the scorer returns one of a fixed set of values; use `OpenEnded` for free-form output typed by a Lightning Type.
`dataType`	string	Required. The data type of the scorer’s output. Valid values: `Text`, `Number`, `LightningType`. Use `Text` or `Number` with `scorerType: Predefined`; use `LightningType` with `scorerType: OpenEnded`.
`lightningType`	string	Required when `dataType` is `LightningType`. The Lightning Type that describes the scorer’s output — for example, `lightning__numberType`, `lightning__textType`, or `lightning__booleanType`.
`semanticType`	string	Optional. Describes how to interpret the scorer’s output for reporting. Valid values: `Dimension`, `Measurement`.
`scorerVersion`	scorerVersion[]	Required. The version configuration for the scorer. Scorers support multiple versions.

scorerVersion

Version numbers must be sequential starting from 1, and each scorer supports a maximum of 100 versions.

Field	Type	Description
`versionNumber`	integer	Required. The version number. Must be sequential starting from `1`.
`status`	string	Required. The lifecycle status. Valid values: `Draft`, `Available`, `Archived`. See Scorer Status.
`description`	string	Required. A description of what the scorer evaluates.
`label`	string	Required. A display label for the scorer version.
`agentAssociation`	AgentAssociation	Required. Associates the scorer with a specific agent.
`engine`	engine[]	Required. The evaluation logic for the scorer.
`outputEnumValue`	outputEnumValue[]	Required. One or more mappings that translate engine output values to pass or fail outcomes.
`specification`	specification[]	Optional. Constraints on the scorer’s output values, such as min, max, step, and threshold.

agentAssociation

Field	Type	Description
`isActive`	boolean	Required. Whether the scorer is active for the associated agent. Can only be `true` for versions with `Available` status. Only one agent association per scorer can have `isActive` set to `true`. Set to `false` for scorers with `engineType: Manual` — manual scorers are annotated by human reviewers and have no active state.
`agentApiName`	string	Required. The API name of the agent. The agent must exist in the org. For example, `Copilot_for_Salesforce`.
`inputScope`	string	Optional. The unit of agent activity that this scorer runs against for the associated agent. Valid values: `Session`, `Intent`. Use `Intent` to score each intent in a session individually.
`samplingRate`	double	Optional. A value greater than `0` and up to `1.0` that controls the sampling rate. Default is `1.0`.

engine

Field	Type	Description
`engineType`	string	Required. The type of evaluation engine. Valid values: `PromptTemplate`, `Manual`. Use `Manual` for scorers that accept human-reviewed scores.
`engineRef`	string	Required when `engineType` is `PromptTemplate`. The API name of the prompt template. Omit for `Manual` scorers.

outputEnumValue

Field	Type	Description
`value`	string	Required. The engine output value that maps to this outcome.
`outcomeType`	string	Optional. The test outcome. Valid values are: `Pass`, `Fail`, `NotApplicable`. Default value is `NotApplicable`.
`description`	string	Optional. A description of what this output value represents.
`isFallback`	boolean	Optional. Whether the LLM can emit this value as the scorer’s fallback — for example, `Neutral` for a `High`/`Normal`/`Low` scorer. Required for `Predefined` scorers with `dataType: Text`; not applicable to `Predefined` scorers with `dataType: Number`.
`isSystemFallback`	boolean	Optional. Whether the system uses this value when the prompt template or LLM execution fails. Often set together with `isFallback` on the same value.

specification

Field	Type	Description
`max`	double	The maximum valid output value.
`min`	double	The minimum valid output value.
`step`	double	The increment between valid output values.
`threshold`	double	Optional. Output values greater than or equal to `threshold` pass.

Scorer Status

Each scorer version progresses through a lifecycle controlled by the status field.

Status	Description
`Draft`	A work in progress. Use `Draft` to edit the scorer’s fields freely. A draft version can’t run evaluations.
`Available`	A finalized scorer. For scorers with `engineType: PromptTemplate`, set `agentAssociation.isActive` to `true` to activate it for evaluation. Manual scorers keep `isActive` set to `false` — reviewers post scores directly through the Annotation API.
`Archived`	A retired version. The Testing API no longer runs evaluations against this version.

Create a scorer version in Draft status when you want to iterate on the scorer’s configuration before finalizing it. When the scorer is ready for use, promote the version to Available. After a version is marked Available, its status can’t be changed back to Draft.

Example Scorer Definition

This example defines a custom scorer that evaluates whether a customer drops off before a conversation resolves. The scorer uses a prompt template to analyze the session and outputs a value of 0 (no drop-off, pass) or 1 (drop-off detected, fail).

1<?xml version="1.0" encoding="UTF-8"?>
2<AiAgentScorerDefinition xmlns="http://soap.sforce.com/2006/04/metadata">
3    <inputScope>Session</inputScope>
4    <dataType>Number</dataType>
5    <scorerVersion>
6        <versionNumber>1</versionNumber>
7        <status>Available</status>
8        <description>Detects whether the customer dropped off before the conversation was resolved.</description>
9        <label>Customer Drop-Off Scorer</label>
10        <agentAssociation>
11            <isActive>true</isActive>
12            <agentApiName>Copilot_for_Salesforce</agentApiName>
13            <inputScope>Session</inputScope>
14        </agentAssociation>
15        <engine>
16            <engineType>PromptTemplate</engineType>
17            <engineRef>Drop_Off_Detection_Template</engineRef>
18        </engine>
19        <outputEnumValue>
20            <isFallback>true</isFallback>
21            <outcomeType>Pass</outcomeType>
22            <value>0</value>
23        </outputEnumValue>
24        <outputEnumValue>
25            <isFallback>false</isFallback>
26            <outcomeType>Fail</outcomeType>
27            <value>1</value>
28        </outputEnumValue>
29        <specification>
30            <valueSpecification>
31                <max>1</max>
32                <min>0</min>
33                <step>1</step>
34                <threshold>1</threshold>
35            </valueSpecification>
36        </specification>
37    </scorerVersion>
38</AiAgentScorerDefinition>

Deploy a Custom Scorer

To deploy a custom scorer, create a project directory with this structure:

1my-scorer-project/
2├── package.xml
3└── aiAgentScorerDefinitions/
4    └── my_scorer_name.aiAgentScorerDefinition

The package.xml file specifies the scorer to deploy:

1<?xml version="1.0" encoding="UTF-8"?>
2<Package xmlns="http://soap.sforce.com/2006/04/metadata">
3    <types>
4        <members>my_scorer_name</members>
5        <name>AiAgentScorerDefinition</name>
6    </types>
7    <version>66.0</version>
8</Package>

The members value must match the filename of your .aiAgentScorerDefinition file (without the extension).

Deploy the scorer to your org with the Salesforce CLI:

1sf project deploy start --metadata-dir my-scorer-project

Deploy a Scorer with a Prompt Template

If your scorer uses the PromptTemplate engine type, you can deploy both the template and the scorer together. Add a genAiPromptTemplates folder containing your prompt template definition, and add the template to package.xml.

1my-scorer-project/
2├── package.xml
3├── genAiPromptTemplates/
4│   └── my_prompt_template.genAiPromptTemplate
5└── aiAgentScorerDefinitions/
6    └── my_scorer_name.aiAgentScorerDefinition

1<?xml version="1.0" encoding="UTF-8"?>
2<Package xmlns="http://soap.sforce.com/2006/04/metadata">
3    <types>
4        <members>my_prompt_template</members>
5        <name>GenAiPromptTemplate</name>
6    </types>
7    <types>
8        <members>my_scorer_name</members>
9        <name>AiAgentScorerDefinition</name>
10    </types>
11    <version>66.0</version>
12</Package>

In package.xml, the GenAiPromptTemplate type must appear before AiAgentScorerDefinition. Metadata API deploys types in the order that they appear, and the prompt template must exist before the scorer that references it can successfully deploy.

Note

Retrieve an Existing Scorer

To retrieve a scorer definition from your org, use the Salesforce CLI:

1sf project retrieve start --metadata AiAgentScorerDefinition:my_scorer_name

You can also retrieve a scorer definition by using the same package.xml that you used for deployment.

Update a Custom Scorer

To update an existing scorer, modify the .aiAgentScorerDefinition file and redeploy. Keep in mind these constraints:

You can add new versions to a scorer, but you can’t delete existing versions.
You can edit a version’s fields while its status is Draft.
You can promote a version’s status from Draft to Available, or move an Available version to Archived. After a version is marked Available, you can’t change its status back to Draft.
You can update the agentAssociation isActive and samplingRate values.
The scorer checks the members name in package.xml. If a scorer with that name already exists, the deployment updates the existing scorer.
Scorers with engineType: Manual don’t have an active state — always set agentAssociation.isActive to false. Human reviewers post scores directly through the Annotation API regardless of the association’s isActive value.