Quickstart
Take a real interaction from a saved trace to your first test run.
On this page
Connect your agent to Shimmer, save a baseline, and run your first regression test. Your agent runs in your own environment; Shimmer manages test suites, traces, and evaluation in your workspace.
Before you begin
You’ll need a Shimmer workspace, Python 3.11 or newer, and your agent’s source code. This guide assumes your project exposes an answer(text) function in support_bot.py and that you have a LangSmith JSON export of a real interaction.
Install the package
In your agent’s Python environment, install Shimmer alongside your existing dependencies:
pip install shimmer-sdk
The package includes the Python SDK and the shimmer CLI. Python imports use shimmer.
Save your first test case
- Open Shimmer and import your LangSmith JSON export using the upload control or drag and drop.
- Open the imported trace and confirm that the user input, agent messages, and tool calls appear.
- Use the suite picker to create a suite named
refundsand add this trace. - In Evaluation criteria, add a success criterion such as “The refund amount matches the original order.”
Choose an interaction that includes a user message. The runner passes that case’s input to your agent.
Point Shimmer at your harness
Create harness.py in your agent’s project:
from shimmer import UserMessage
from support_bot import answer
def run(message: UserMessage) -> None:
answer(message.text)
Your answer function needs LangSmith instrumentation, such as a @traceable root function and a traced model client. See LangSmith for a complete example.
Run locally
Keep your agent’s model and tool credentials available in your environment. From the directory containing harness.py, run:
LANGSMITH_TRACING=local shimmer --path harness --test-suite refunds
--path harness points to the importable module containing your run function. LANGSMITH_TRACING=local captures the interaction for Shimmer without uploading it to LangSmith.
Open Test runs, select the new run, and compare the candidate against its baseline. Review the assertions to understand whether the behavior met your criteria.
Run on pull requests
Use the same harness with the Shimmer GitHub Action. Connect your GitHub repository to your Shimmer workspace, grant the job id-token: write, and point the Action at your harness and suite. The Action authenticates with GitHub OIDC.