Skip to content
shimmer/docs
Run and compare

Run and compare

Inspect execution, evaluation results, and the differences from your baseline.

On this page

Run your agent locally or on a GitHub runner, then review the results in Shimmer. Each run connects your current agent’s behavior to the baselines saved in your workspace.

Run a suite locally

With the package installed and your harness available, execute:

terminal
LANGSMITH_TRACING=local shimmer --path harness --test-suite refunds

Shimmer creates a suite run, your harness executes each case, and the collected traces are sent to the hosted platform for evaluation. Cases run sequentially.

Run on pull requests

Use the Shimmer GitHub Action to run the same harness and suite in CI. Grant the job id-token: write for GitHub OIDC authentication and install your agent’s dependencies in the workflow.

The agent executes on your GitHub runner. Shimmer manages the suite, collected traces, and evaluation results in your workspace.

Open the run

Open Test runs, select your suite run, and inspect the case results. Open a case to compare the baseline and candidate alongside its assertions.

Start with the first meaningful difference. A changed assistant sentence may be harmless; an unexpected tool call or argument can change the outcome of the interaction.

A small change. A very different outcome.EXAMPLE
BASELINE

Issue a refund for my $42 order.

refund_order
{ "amount": 42 } Expected amount
CANDIDATE

Issue a refund for my $42 order.

refund_order
{ "amount": 420 }Changed tool argument

Separate execution from evaluation

Execution status describes whether the agent ran and its trace was saved. Evaluation status describes whether the recorded behavior met the baseline’s criteria.

  • success execution: the callback completed and a candidate trace was saved.
  • fail execution: the callback, collection, or upload failed. Inspect the error text.
  • failure evaluation: the criteria check found behavior that did not meet expectations.
  • inconclusive evaluation: the evaluator could not make a decisive judgment.
  • error evaluation: the check could not complete.
  • not_evaluated: no completed criteria verdict is available.

Use the result in automation

The CLI exits with a nonzero status when execution fails or the final evaluation verdict is not success or not_evaluated. A failing suite also fails the GitHub Action step.

After investigating a regression, fix the agent and re-run the same suite. Keep the case so the behavior remains covered.