Write a Python harness
Connect your agent to the test runner with one small Python function.
On this page
A harness is the small adapter between a Shimmer test case and your agent. The runner imports a module, finds its run function, and calls it once per case.
Define the entry point
Create harness.py in your Python project:
from shimmer import UserMessage
from support_bot import answer
def run(message: UserMessage) -> None:
answer(message.text)
UserMessage.text contains the test case input. The callback’s return value is ignored: Shimmer uses the collected trace to inspect and evaluate the result.
Your agent must produce LangSmith-instrumented activity during the callback. See LangSmith for an example using traceable and wrap_openai.
Run the harness
Run from a directory where harness can be imported:
LANGSMITH_TRACING=local shimmer \
--path harness \
--test-suite refunds
--path is a Python module name, not a filesystem path. For tests/harness.py in an importable package, use --path tests.harness.
Call an asynchronous agent
The current runner calls a synchronous callback and does not await its result. Adapt an async agent with a synchronous entry point when running from the CLI:
import asyncio
from shimmer import UserMessage
from support_bot import answer_async
def run(message: UserMessage) -> None:
asyncio.run(answer_async(message.text))
Use this pattern from a normal CLI process without an existing event loop.
Use the same harness in CI
Commit the harness to your agent’s repository, then point the Shimmer GitHub Action at the same module:
permissions:
contents: read
id-token: write
steps:
- name: Run Shimmer suite
uses: getshimmer/run-tests@v1
with:
harness: harness
test-suite: refunds
env:
LANGSMITH_TRACING: local
This example shows the job permissions and Shimmer step. Connect your repository to your Shimmer workspace so the Action can authenticate with GitHub OIDC. Install your agent’s dependencies before this step and provide its model and tool credentials in the step’s environment. The Action runs your harness on the GitHub runner and sends the results to your Shimmer workspace.