Skip to content
shimmer/docs
Reference

CLI and configuration

Install the package, connect your workspace, and run a suite from your terminal.

On this page

Install the Shimmer package in your agent’s environment, connect your workspace, and point the CLI at your harness. Results are available in the Shimmer web app.

Install

terminal
pip install shimmer-sdk

Python 3.11 or newer is required. Install your agent’s dependencies into the same environment so the CLI can import the harness and execute your code.

GitHub Actions authentication

The Shimmer Action authenticates using GitHub OpenID Connect (OIDC). Connect your repository to your Shimmer workspace and grant the test job id-token: write. The Action handles the short-lived token automatically.

See GitHub Actions for the complete workflow.

Command

terminal
LANGSMITH_TRACING=local shimmer --path harness --test-suite refunds

You can also use the Python module entry point:

terminal
LANGSMITH_TRACING=local python -m shimmer --path harness --test-suite refunds

Both commands run your agent locally and send its collected trace to Shimmer for evaluation and comparison.

Arguments

--path MODULE — required. An importable Python module that exposes run(message). Use harness for harness.py, or tests.harness for a module in an importable tests package. Run from your project root and omit the .py extension.

--test-suite NAME — required. The exact name of a suite in your Shimmer workspace. Suite names are case-sensitive. Use a unique name so the selection is unambiguous.

--help. Print usage and argument descriptions.

Environment variables

  • LANGSMITH_TRACING=local captures instrumented activity without uploading it to LangSmith. The trace is still sent to Shimmer.
  • LANGSMITH_TRACING=true also enables LangSmith uploads; supply your LangSmith credentials if you choose this mode.
  • Your agent’s provider and tool credentials, such as OPENAI_API_KEY, must be available to the process running the harness.

Shimmer manages evaluation in the hosted platform. Your agent uses its own configured model and tool integrations.

Exit behavior

A successful run exits with status 0. Execution failures, evaluation failures, inconclusive verdicts, and evaluation errors produce a nonzero exit status.

The runner also accepts not_evaluated when execution succeeded. Add criteria to your baseline cases so a completed run checks the behavior you care about.

To run the same harness in CI, follow GitHub Actions.