Skip to content
shimmer/docs
Get started

Quickstart

Take a real interaction from a saved trace to your first test run.

On this page

Connect your agent to Shimmer, save a baseline, and run your first regression test. Your agent runs in your own environment; Shimmer manages test suites, traces, and evaluation in your workspace.

Before you begin

You’ll need a Shimmer workspace, Python 3.11 or newer, and your agent’s source code. This guide assumes your project exposes an answer(text) function in support_bot.py and that you have a LangSmith JSON export of a real interaction.

Install the package

In your agent’s Python environment, install Shimmer alongside your existing dependencies:

terminal
pip install shimmer-sdk

The package includes the Python SDK and the shimmer CLI. Python imports use shimmer.

Save your first test case

  1. Open Shimmer and import your LangSmith JSON export using the upload control or drag and drop.
  2. Open the imported trace and confirm that the user input, agent messages, and tool calls appear.
  3. Use the suite picker to create a suite named refunds and add this trace.
  4. In Evaluation criteria, add a success criterion such as “The refund amount matches the original order.”

Choose an interaction that includes a user message. The runner passes that case’s input to your agent.

Point Shimmer at your harness

Create harness.py in your agent’s project:

python
from shimmer import UserMessage
from support_bot import answer


def run(message: UserMessage) -> None:
    answer(message.text)

Your answer function needs LangSmith instrumentation, such as a @traceable root function and a traced model client. See LangSmith for a complete example.

Run locally

Keep your agent’s model and tool credentials available in your environment. From the directory containing harness.py, run:

terminal
LANGSMITH_TRACING=local shimmer --path harness --test-suite refunds

--path harness points to the importable module containing your run function. LANGSMITH_TRACING=local captures the interaction for Shimmer without uploading it to LangSmith.

Open Test runs, select the new run, and compare the candidate against its baseline. Review the assertions to understand whether the behavior met your criteria.

Run on pull requests

Use the same harness with the Shimmer GitHub Action. Connect your GitHub repository to your Shimmer workspace, grant the job id-token: write, and point the Action at your harness and suite. The Action authenticates with GitHub OIDC.