Skip to content
shimmer/docs
Build test suites

Evaluation criteria

Describe the behavior you expect, and the failures you want to catch.

On this page

Evaluation criteria describe the behavior you want from your agent. They belong to the baseline trace and are used to evaluate the candidate trace during a test run.

Add a criterion

Open a trace and find the Evaluation criteria panel. Add a statement under the appropriate group:

  • Success criteria: behavior that should occur.
  • Failure criteria: behavior that should not occur.

For a refund interaction, write a success criterion such as “The agent refunds exactly the amount of the original order.” Add a failure criterion such as “The agent issues a refund without confirming the order.”

Write observable statements

Keep each criterion focused on one behavior. Include the context needed to judge it from the trace.

“The agent is helpful” leaves too much open to interpretation. “The agent tells the user that the order is outside the 30-day refund window” describes a specific outcome.

Separate independent expectations into separate criteria. That makes a failed assertion easier to investigate.

Read assertion results

After a case runs, open its result to inspect each assertion, its status, and the reason provided by the evaluator.

A changed conversation is not automatically a failure. Use the comparison to understand what changed and the assertions to understand whether the behavior met your criteria.

inconclusive means the evaluator did not reach a decisive verdict. error means evaluation could not complete. Investigate these results before treating the test as a pass.

Managed evaluation

Shimmer evaluates the candidate trace against your baseline’s criteria in the hosted platform. Define the behavior you expect in your workspace, then run your harness locally or through GitHub Actions. Shimmer manages the evaluation service and its credentials.

Continue with Run and compare to understand execution and suite verdicts.