Skip to content
shimmer/docs
Get started

Introduction

Turn real agent traces into regression tests. Understand what changed before you ship.

On this page

Shimmer helps you answer a simple question: did this change make my agent better? Save real interactions as test cases, re-run your agent, and inspect the behavior that changed. Your agent runs locally or in GitHub Actions; Shimmer manages your suites, traces, and evaluation in the cloud.

01CaptureA real interaction
02CurateYour test suite
03RunYour changed agent
04CompareBehavior, explained

Start building

Whether you already have traces or are starting with your agent’s code, there’s a short path to your first test.

From a trace to a test

A useful regression test starts with a real interaction. An agent receives a request, makes decisions, calls tools, and returns an answer. Shimmer keeps that interaction as a baseline so you can compare the next version against it.

The final answer is only part of the story. A changed tool argument can matter more than a rewritten sentence.

A small change. A very different outcome.EXAMPLE
BASELINE

Issue a refund for my $42 order.

refund_order
{ "amount": 42 } Expected amount
CANDIDATE

Issue a refund for my $42 order.

refund_order
{ "amount": 420 }Changed tool argument

Add evaluation criteria to describe what should happen: “The refund amount matches the original order.” Run the suite after changing your agent, then inspect both the evaluation result and the recorded behavior.

The Shimmer workflow

  1. Capture. Import a LangSmith trace or collect one while running a test harness.
  2. Curate. Group representative interactions into a named test suite.
  3. Run. Execute your current agent against the suite’s inputs.
  4. Compare. Read the results, inspect changed messages and tool calls, and investigate failures.

Each case links a baseline interaction to a candidate execution. A suite gives you a repeatable collection of these comparisons.

Choose your next step