Introduction
Turn real agent traces into regression tests. Understand what changed before you ship.
On this page
Shimmer helps you answer a simple question: did this change make my agent better? Save real interactions as test cases, re-run your agent, and inspect the behavior that changed. Your agent runs locally or in GitHub Actions; Shimmer manages your suites, traces, and evaluation in the cloud.
Start building
Whether you already have traces or are starting with your agent’s code, there’s a short path to your first test.
Connect your Python agent, create a suite, and inspect a test run.
Bring your own tracesImport a LangSmith export and turn real interactions into test cases.
From a trace to a test
A useful regression test starts with a real interaction. An agent receives a request, makes decisions, calls tools, and returns an answer. Shimmer keeps that interaction as a baseline so you can compare the next version against it.
The final answer is only part of the story. A changed tool argument can matter more than a rewritten sentence.
Issue a refund for my $42 order.
{ "amount": 42 } Expected amountIssue a refund for my $42 order.
{ "amount": 420 }Changed tool argumentAdd evaluation criteria to describe what should happen: “The refund amount matches the original order.” Run the suite after changing your agent, then inspect both the evaluation result and the recorded behavior.
The Shimmer workflow
- Capture. Import a LangSmith trace or collect one while running a test harness.
- Curate. Group representative interactions into a named test suite.
- Run. Execute your current agent against the suite’s inputs.
- Compare. Read the results, inspect changed messages and tool calls, and investigate failures.
Each case links a baseline interaction to a candidate execution. A suite gives you a repeatable collection of these comparisons.