Skip to content
shimmer/docs
Reference

Troubleshooting

Resolve common setup, tracing, suite lookup, and evaluation issues.

On this page

Start by distinguishing a connection issue, an agent execution failure, and an evaluation result. The CLI output and the case’s errors and assertions usually identify the next step.

Shimmer cannot connect to your workspace

Check that your network permits outbound HTTPS connections to Shimmer.

In GitHub Actions, confirm that your repository is connected to the workspace containing your suite and that the test job grants id-token: write. The Shimmer Action uses GitHub OIDC to authenticate. See GitHub Actions for a complete workflow.

No trace collector is enabled

Enable LangSmith tracing before calling the runner:

terminal
LANGSMITH_TRACING=local shimmer --path harness --test-suite refunds

Your callback still needs to execute instrumented code. Enabling collection does not automatically instrument an arbitrary agent.

The suite cannot be found

Check the exact suite name in Test suites. Names are case-sensitive. Use a unique name; multiple matching suites cause an error.

For GitHub Actions runs, confirm that your repository is connected to the workspace containing the suite. Use the same name in the CLI’s --test-suite argument and the Action’s test-suite input.

The harness cannot be imported

Pass a module name, such as harness or tests.harness, without the .py extension. Run from your agent’s project root and confirm that the module exposes run.

terminal
shimmer --path tests.harness --test-suite refunds

Install your agent’s dependencies in the same Python environment as Shimmer. In GitHub Actions, ensure the harness is committed and the workflow installs those dependencies before the Shimmer step.

For asynchronous agents, use a synchronous adapter that runs the coroutine. The runner does not await an async callback.

The candidate trace is missing

Inspect the execution error for callback, collection, or upload failures. Confirm that your instrumented agent runs inside the harness callback and that your network can reach Shimmer.

For imported baselines, include the full LangSmith interaction and child runs. For collected candidates, start with the instrumentation example in LangSmith.

Evaluation failed or did not complete

Open the case’s assertions to distinguish a failed criterion from an evaluation error. Review the recorded interaction and the criterion’s wording to confirm that the trace contains the evidence needed to judge it.

An inconclusive verdict needs review. A not_evaluated verdict is accepted by the runner, but does not show that your expected behavior was checked. Add or review the baseline’s evaluation criteria.

Evaluation runs in Shimmer. If an evaluation error persists, keep the run ID and contact Shimmer support through your workspace.

The GitHub Action cannot authenticate

Grant the test job id-token: write and confirm that its repository is connected to the correct Shimmer workspace. The Action requests a short-lived GitHub OIDC token and Shimmer checks the workflow’s identity. Check that the repository, branch, and event are allowed by your workspace’s repository connection.

Your agent’s model and tool credentials are separate from Shimmer authentication. Supply those through the step’s environment. Ordinary pull request workflows from forks do not receive repository secrets, so runs that depend on provider credentials need an appropriate trusted context. See GitHub Actions.

The OpenAI shim fails outside a test

The shim requires an active Shimmer case run. Use a regular traced OpenAI client for ordinary application calls, and apply the Shimmer shim for test execution. See OpenAI for supported operations.