CLI and configuration
Install the package, connect your workspace, and run a suite from your terminal.
On this page
Install the Shimmer package in your agent’s environment, connect your workspace, and point the CLI at your harness. Results are available in the Shimmer web app.
Install
pip install shimmer-sdk
Python 3.11 or newer is required. Install your agent’s dependencies into the same environment so the CLI can import the harness and execute your code.
GitHub Actions authentication
The Shimmer Action authenticates using GitHub OpenID Connect (OIDC). Connect your repository to your Shimmer workspace and grant the test job id-token: write. The Action handles the short-lived token automatically.
See GitHub Actions for the complete workflow.
Command
LANGSMITH_TRACING=local shimmer --path harness --test-suite refunds
You can also use the Python module entry point:
LANGSMITH_TRACING=local python -m shimmer --path harness --test-suite refunds
Both commands run your agent locally and send its collected trace to Shimmer for evaluation and comparison.
Arguments
--path MODULE — required. An importable Python module that exposes run(message). Use harness for harness.py, or tests.harness for a module in an importable tests package. Run from your project root and omit the .py extension.
--test-suite NAME — required. The exact name of a suite in your Shimmer workspace. Suite names are case-sensitive. Use a unique name so the selection is unambiguous.
--help. Print usage and argument descriptions.
Environment variables
LANGSMITH_TRACING=localcaptures instrumented activity without uploading it to LangSmith. The trace is still sent to Shimmer.LANGSMITH_TRACING=truealso enables LangSmith uploads; supply your LangSmith credentials if you choose this mode.- Your agent’s provider and tool credentials, such as
OPENAI_API_KEY, must be available to the process running the harness.
Shimmer manages evaluation in the hosted platform. Your agent uses its own configured model and tool integrations.
Exit behavior
A successful run exits with status 0. Execution failures, evaluation failures, inconclusive verdicts, and evaluation errors produce a nonzero exit status.
The runner also accepts not_evaluated when execution succeeded. Add criteria to your baseline cases so a completed run checks the behavior you care about.
To run the same harness in CI, follow GitHub Actions.