Skip to content
shimmer/docs
Integrations

LangSmith

Use LangSmith instrumentation to collect agent traces during a Shimmer test run.

On this page

Shimmer uses LangSmith instrumentation to capture the activity inside a Python test callback. You can also import existing LangSmith JSON exports through the website.

Instrument your agent

Use a traced root function and a traced model client. This example shows the structure of support_bot.py; configure your provider credentials in the process environment.

python
from langsmith import traceable
from langsmith.wrappers import wrap_openai
from openai import OpenAI

client = wrap_openai(OpenAI())


@traceable
def answer(text: str) -> str:
    response = client.chat.completions.create(
        model="your-model-id",
        messages=[{"role": "user", "content": text}],
    )
    return response.choices[0].message.content or ""

Replace your-model-id with a model available to your provider account. Your harness can now call answer(message.text) while Shimmer collects its activity.

Capture without uploading to LangSmith

Enable local capture before starting the CLI:

terminal
LANGSMITH_TRACING=local shimmer \
  --path harness \
  --test-suite refunds

Shimmer enables instrumentation inside its collection context while leaving LangSmith uploads disabled. The candidate trace is still sent to your Shimmer workspace.

Upload to both services

Set LANGSMITH_TRACING=true and configure your LangSmith credentials to upload runs to LangSmith as well as Shimmer.

The collector honors the current LangSmith tracing context. Disabling tracing without enabling another collector causes the runner to report that no trace collector is enabled.

Import existing traces

Export a complete interaction from LangSmith as JSON, then import it into Shimmer. Include the child model and tool runs so the normalized conversation contains the relevant events.

Importing existing traces and collecting new test runs are separate paths: importing creates a source trace you can add to a suite; running a harness creates candidate traces for that suite’s cases.