Evaluation criteria
Describe the behavior you expect, and the failures you want to catch.
On this page
Evaluation criteria describe the behavior you want from your agent. They belong to the baseline trace and are used to evaluate the candidate trace during a test run.
Add a criterion
Open a trace and find the Evaluation criteria panel. Add a statement under the appropriate group:
- Success criteria: behavior that should occur.
- Failure criteria: behavior that should not occur.
For a refund interaction, write a success criterion such as “The agent refunds exactly the amount of the original order.” Add a failure criterion such as “The agent issues a refund without confirming the order.”
Write observable statements
Keep each criterion focused on one behavior. Include the context needed to judge it from the trace.
“The agent is helpful” leaves too much open to interpretation. “The agent tells the user that the order is outside the 30-day refund window” describes a specific outcome.
Separate independent expectations into separate criteria. That makes a failed assertion easier to investigate.
Read assertion results
After a case runs, open its result to inspect each assertion, its status, and the reason provided by the evaluator.
A changed conversation is not automatically a failure. Use the comparison to understand what changed and the assertions to understand whether the behavior met your criteria.
inconclusive means the evaluator did not reach a decisive verdict. error means evaluation could not complete. Investigate these results before treating the test as a pass.
Managed evaluation
Shimmer evaluates the candidate trace against your baseline’s criteria in the hosted platform. Define the behavior you expect in your workspace, then run your harness locally or through GitHub Actions. Shimmer manages the evaluation service and its credentials.
Continue with Run and compare to understand execution and suite verdicts.