Capture and inspect traces
Bring real agent behavior into Shimmer and follow every message and tool call.
On this page
Start with real interactions that represent the behavior you want to protect. Include ordinary successes, difficult inputs, and known failures.
Import a LangSmith export
Open Traces and use the upload control, or drag a LangSmith JSON export onto the trace list. Shimmer imports the payload and shows a summary in the list.
Import the complete interaction, including the child runs that contain model and tool activity. Missing children can leave the trace without the context needed to understand what happened.
Each upload creates a new Shimmer trace ID. Re-importing an export does not update the previous import.
Inspect an interaction
Open a trace to read its ordered events. Follow the user request through the assistant’s decisions, tool calls, returned results, and final answer.
Check more than the final response. Tool names, arguments, and results often explain why an apparently reasonable answer hides a regression.
Issue a refund for my $42 order.
{ "amount": 42 } Expected amountIssue a refund for my $42 order.
{ "amount": 420 }Changed tool argumentCollect traces from a test run
The Python runner collects LangSmith-instrumented activity during your harness callback and uploads the result to Shimmer.
With the package installed, enable tracing before starting the CLI:
LANGSMITH_TRACING=local shimmer \
--path harness \
--test-suite refunds
This captures local traces without uploading them to LangSmith. See the LangSmith integration for a complete instrumentation example.
Turn a trace into a case
Use the suite picker to add the trace to an existing suite or create a new suite. Choose a meaningful name and add evaluation criteria to the source trace.
Removing suite membership keeps the original trace. Deleting a source trace can make historical comparisons that reference it unavailable, so keep baselines you still need.