Run and compare
Inspect execution, evaluation results, and the differences from your baseline.
On this page
Run your agent locally or on a GitHub runner, then review the results in Shimmer. Each run connects your current agent’s behavior to the baselines saved in your workspace.
Run a suite locally
With the package installed and your harness available, execute:
LANGSMITH_TRACING=local shimmer --path harness --test-suite refunds
Shimmer creates a suite run, your harness executes each case, and the collected traces are sent to the hosted platform for evaluation. Cases run sequentially.
Run on pull requests
Use the Shimmer GitHub Action to run the same harness and suite in CI. Grant the job id-token: write for GitHub OIDC authentication and install your agent’s dependencies in the workflow.
The agent executes on your GitHub runner. Shimmer manages the suite, collected traces, and evaluation results in your workspace.
Open the run
Open Test runs, select your suite run, and inspect the case results. Open a case to compare the baseline and candidate alongside its assertions.
Start with the first meaningful difference. A changed assistant sentence may be harmless; an unexpected tool call or argument can change the outcome of the interaction.
Issue a refund for my $42 order.
{ "amount": 42 } Expected amountIssue a refund for my $42 order.
{ "amount": 420 }Changed tool argumentSeparate execution from evaluation
Execution status describes whether the agent ran and its trace was saved. Evaluation status describes whether the recorded behavior met the baseline’s criteria.
successexecution: the callback completed and a candidate trace was saved.failexecution: the callback, collection, or upload failed. Inspect the error text.failureevaluation: the criteria check found behavior that did not meet expectations.inconclusiveevaluation: the evaluator could not make a decisive judgment.errorevaluation: the check could not complete.not_evaluated: no completed criteria verdict is available.
Use the result in automation
The CLI exits with a nonzero status when execution fails or the final evaluation verdict is not success or not_evaluated. A failing suite also fails the GitHub Action step.
After investigating a regression, fix the agent and re-run the same suite. Keep the case so the behavior remains covered.