Core concepts
The building blocks of a Shimmer test: traces, suites, baselines, and runs.
On this page
Shimmer connects a recorded interaction to a repeatable test. These are the concepts you’ll see in the SDK and interface.
Trace
A trace records an agent interaction as ordered events: user messages, assistant messages, tool calls, and tool results. Provider payloads are normalized so you can inspect the conversation and tool behavior in one place.
Importing a trace creates a Shimmer trace ID. Import the same provider payload again and you get another trace; provider IDs do not deduplicate imports.
Baseline
The source trace attached to a test case is its baseline. It gives you a concrete example to compare against, and its evaluation criteria describe the behavior you want from future executions.
A baseline helps you understand a change. It does not mean every word or tool argument must remain identical: your criteria define the behavior that matters.
Test case and suite
A test case adds a source trace to a named test suite. The suite groups interactions you want to run together, such as refund handling or account recovery.
The Python runner selects a suite by its exact name. Keep suite names unique so the selection is unambiguous.
Removing a trace from a suite removes its membership; it does not delete the original trace.
Suite run and case run
A suite run groups the case executions for one test run. Each case run records its input, execution status, candidate trace, and evaluation result.
The Python runner executes cases sequentially. Your callback receives a UserMessage for each case, and Shimmer collects the traces produced during that callback.
Execution and evaluation
Execution status tracks whether the agent ran and its trace was saved: pending, in_progress, success, or fail.
Evaluation status tracks the criteria check: not_evaluated, in_progress, success, failure, inconclusive, or error.
A case can execute successfully and fail evaluation. An evaluation error means the check could not complete; it is different from a finding that the agent violated a criterion.
Learn how these results appear in Run and compare.