Skip to content
shimmer/docs
Run and compare

GitHub Actions

Point the Shimmer Action at your harness and run regression tests on every pull request.

On this page

Run your Shimmer test suites on pull requests using the same harness you use locally. Your agent executes on your GitHub runner; Shimmer stores the traces and evaluates the results in your workspace.

Before you begin

Complete the quickstart, commit your harness to your own repository, and confirm that the suite runs locally. The examples use a module named harness and a suite named refunds.

Connect your repository

Connect your GitHub repository to your Shimmer workspace. The Action authenticates using GitHub OpenID Connect (OIDC): it requests a short-lived identity token for the workflow, and Shimmer verifies that the workflow belongs to a connected repository.

Grant the test job id-token: write so the Action can request that token. The Action handles authentication automatically.

Your agent’s model and tool credentials are separate. Add the credentials your agent needs as GitHub Actions secrets. The example below uses OPENAI_API_KEY; replace or extend it for your agent.

Add the workflow

Create .github/workflows/shimmer.yml in your agent’s repository:

yaml
name: Agent regression tests

on:
  pull_request:
  workflow_dispatch:

jobs:
  shimmer:
    runs-on: ubuntu-latest
    permissions:
      contents: read
      id-token: write
    steps:
      - uses: actions/checkout@v6

      - uses: actions/setup-python@v6
        with:
          python-version: "3.11"

      - name: Install agent dependencies
        run: pip install -r requirements.txt

      - name: Run Shimmer suite
        uses: getshimmer/run-tests@v1
        with:
          harness: harness
          test-suite: refunds
        env:
          LANGSMITH_TRACING: local
          OPENAI_API_KEY: ${{ secrets.OPENAI_API_KEY }}

Replace the dependency installation step with your project’s usual installation command. Set harness to the same module name you pass to the CLI’s --path argument and test-suite to the exact suite name in Shimmer.

The Shimmer Action installs the Shimmer package and runs the selected suite. Your workflow supplies Python, your agent’s dependencies, and its provider credentials.

Action inputs

harness — required. The importable module containing run(message). Use harness for harness.py, or tests.harness for a module in a package. The module must be importable from the repository root.

test-suite — required. The exact name of the suite to execute in your Shimmer workspace.

Use env on the Action step for tracing configuration and the model or tool credentials your agent needs.

Workflow permissions

contents: read allows the checkout step to read your agent’s repository. id-token: write allows the Shimmer Action to request a GitHub OIDC token. Shimmer uses that identity to authorize the run against the connected workspace.

See GitHub’s OIDC documentation for how token permissions work.

Review the results

The workflow runs on pull requests and can also be started manually from the Actions tab. A failing suite fails the Action step. Open Test runs in Shimmer to inspect individual cases, evaluation assertions, and baseline comparisons.

To require the job before merging, add the shimmer job as a required status check in your repository’s branch protection settings.

Troubleshoot a workflow

Confirm that the harness and all of its dependencies are installed in the selected Python environment. Verify the suite name, the repository’s workspace connection, and the job’s id-token: write permission. Check that your agent’s provider secrets are available to the job.

GitHub does not provide repository secrets to ordinary pull request workflows from forks. Run those tests in a trusted context with the required secrets rather than changing the workflow to execute untrusted code with credentials.

For import errors, missing traces, and evaluation results, see Troubleshooting.