Skip to content
shimmer/docs
Run and compare

Write a Python harness

Connect your agent to the test runner with one small Python function.

On this page

A harness is the small adapter between a Shimmer test case and your agent. The runner imports a module, finds its run function, and calls it once per case.

Define the entry point

Create harness.py in your Python project:

python
from shimmer import UserMessage
from support_bot import answer


def run(message: UserMessage) -> None:
    answer(message.text)

UserMessage.text contains the test case input. The callback’s return value is ignored: Shimmer uses the collected trace to inspect and evaluate the result.

Your agent must produce LangSmith-instrumented activity during the callback. See LangSmith for an example using traceable and wrap_openai.

Run the harness

Run from a directory where harness can be imported:

terminal
LANGSMITH_TRACING=local shimmer \
  --path harness \
  --test-suite refunds

--path is a Python module name, not a filesystem path. For tests/harness.py in an importable package, use --path tests.harness.

Call an asynchronous agent

The current runner calls a synchronous callback and does not await its result. Adapt an async agent with a synchronous entry point when running from the CLI:

python
import asyncio

from shimmer import UserMessage
from support_bot import answer_async


def run(message: UserMessage) -> None:
    asyncio.run(answer_async(message.text))

Use this pattern from a normal CLI process without an existing event loop.

Use the same harness in CI

Commit the harness to your agent’s repository, then point the Shimmer GitHub Action at the same module:

yaml
permissions:
  contents: read
  id-token: write

steps:
  - name: Run Shimmer suite
    uses: getshimmer/run-tests@v1
    with:
      harness: harness
      test-suite: refunds
    env:
      LANGSMITH_TRACING: local

This example shows the job permissions and Shimmer step. Connect your repository to your Shimmer workspace so the Action can authenticate with GitHub OIDC. Install your agent’s dependencies before this step and provide its model and tool credentials in the step’s environment. The Action runs your harness on the GitHub runner and sends the results to your Shimmer workspace.