Skip to content
shimmer/docs
Capture traces

Capture and inspect traces

Bring real agent behavior into Shimmer and follow every message and tool call.

On this page

Start with real interactions that represent the behavior you want to protect. Include ordinary successes, difficult inputs, and known failures.

Import a LangSmith export

Open Traces and use the upload control, or drag a LangSmith JSON export onto the trace list. Shimmer imports the payload and shows a summary in the list.

Import the complete interaction, including the child runs that contain model and tool activity. Missing children can leave the trace without the context needed to understand what happened.

Each upload creates a new Shimmer trace ID. Re-importing an export does not update the previous import.

Inspect an interaction

Open a trace to read its ordered events. Follow the user request through the assistant’s decisions, tool calls, returned results, and final answer.

Check more than the final response. Tool names, arguments, and results often explain why an apparently reasonable answer hides a regression.

A small change. A very different outcome.EXAMPLE
BASELINE

Issue a refund for my $42 order.

refund_order
{ "amount": 42 } Expected amount
CANDIDATE

Issue a refund for my $42 order.

refund_order
{ "amount": 420 }Changed tool argument

Collect traces from a test run

The Python runner collects LangSmith-instrumented activity during your harness callback and uploads the result to Shimmer.

With the package installed, enable tracing before starting the CLI:

terminal
LANGSMITH_TRACING=local shimmer \
  --path harness \
  --test-suite refunds

This captures local traces without uploading them to LangSmith. See the LangSmith integration for a complete instrumentation example.

Turn a trace into a case

Use the suite picker to add the trace to an existing suite or create a new suite. Choose a meaningful name and add evaluation criteria to the source trace.

Removing suite membership keeps the original trace. Deleting a source trace can make historical comparisons that reference it unavailable, so keep baselines you still need.