Python SDK
Install the Shimmer package and connect your Python agent to your workspace.
On this page
Install shimmer-sdk from your Python package manager. The package provides the shimmer CLI, the UserMessage harness input, and integration tools for collecting your agent’s behavior.
Install the package
pip install shimmer-sdk
Python 3.11 or newer is required. Install your agent’s dependencies in the same environment.
Harness input
Your harness exposes a synchronous run function that accepts a UserMessage:
from shimmer import UserMessage
from support_bot import answer
def run(message: UserMessage) -> None:
answer(message.text)
message.text contains the input from a test case. The return value is ignored; Shimmer collects the instrumented interaction and evaluates that trace.
Point the CLI at this module with --path harness, or set the GitHub Action’s harness input to harness. See Write a Python harness for asynchronous agents and module layout.
Test execution
Use the CLI or the Shimmer GitHub Action to execute a suite. Both invoke your harness once per case and handle suite lookup, execution, and trace collection for you.
Execution and evaluation are separate: your agent can run successfully and still fail a behavioral criterion. Review the run and its assertions in your Shimmer workspace.
The runner reports missing or ambiguous suites, disabled tracing, callback exceptions, collection failures, and unsuccessful evaluation verdicts. See Run and compare for how to read results.
Data models
UserMessage is the input to your harness. Read message.text to get the case’s user input and pass it to your agent.
The model is frozen. Treat the input as immutable; keep agent-specific setup and state inside your harness or application.
Integrations
Use LangSmith instrumentation to capture the activity inside your harness. Set LANGSMITH_TRACING=local to collect for Shimmer without also uploading to LangSmith.
For recorded tool-result preparation with the synchronous OpenAI client, import shim_openai from shimmer.openai.