Skip to content
shimmer/docs
Build test suites

Create a test suite

Build a focused regression suite from interactions you want your agent to get right.

On this page

A test suite is a named collection of interactions you want your agent to handle reliably. Start small enough that you can explain what every case protects.

Create a suite

Open Test suites and create a suite, or create one through a trace’s suite picker. Give it a specific name such as refunds or account-recovery.

The runner looks up the exact suite name. refunds and Refunds are different names; duplicate matches cause an error instead of selecting one arbitrarily.

Add cases from traces

Open Traces, find an interaction, and use the suite picker to add it. Each case keeps the selected source trace as its baseline.

For a refund suite, consider interactions that cover:

  • A normal refund for an eligible order.
  • An order outside the refund window.
  • A user requesting an amount that differs from the order.
  • A tool failure that the agent should explain clearly.

Define expected behavior

Open each source trace and add evaluation criteria. Use observable statements that a reviewer can judge from the interaction.

For example, a success criterion could be “The agent confirms the original order amount before issuing a refund.” A failure criterion could be “The agent refunds more than the original order amount.”

Maintain a useful suite

Add a case when you find a meaningful new behavior or regression. Prefer distinct failure modes over many copies of the same interaction.

Removing a trace from a suite does not delete the trace. Keep baseline traces available while you still need their comparisons and history.

Once the suite contains cases, connect your agent with a Python harness.