Skip to content

CI integration

GitHub Actions

name: Evals
on: [pull_request]
permissions:
  contents: read
jobs:
  evals:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v5
      - uses: actions/setup-python@v6
        with:
          python-version: "3.12"
      - run: pip install sic-evalkit
      - run: |
          mkdir -p evalkit-results
          evalkit run evals.yaml \
            --output evalkit-results/results.json \
            --junit evalkit-results/junit.xml \
            --markdown evalkit-results/summary.md \
            --badge evalkit-results/badge.json
        env:
          OPENAI_API_KEY: ${{ secrets.OPENAI_API_KEY }}
      - if: always()
        run: cat evalkit-results/summary.md >> "$GITHUB_STEP_SUMMARY"

The run step writes results.json, junit.xml, summary.md, and badge.json to evalkit-results, and the last step appends the Markdown summary to the job summary. The step fails when the suite fails, which fails the job. Add any other evalkit run arguments, such as --tag smoke, --fail-under, or --baseline and --threshold, to the same command.

Baselines in CI

A simple pattern: upload results.json as an artifact on main, download the latest one in pull request workflows, and pass it with --baseline. The response cache makes the main-branch run cheap, and you can cache .evalkit/ between jobs with actions/cache to skip repeat API calls.

Other CI systems

Run the CLI and publish the JUnit file with your system's test report feature. Install with pip:

pip install sic-evalkit
evalkit run evals.yaml --junit evalkit-junit.xml -o results.json

Badge

--badge badge.json writes a shields.io endpoint badge such as {"schemaVersion": 1, "label": "evals", "message": "7/8 passed", "color": "yellow"}. Publish the file anywhere public, such as GitHub Pages or a gist, and point https://img.shields.io/endpoint?url=<url-of-badge.json> at it.

Results format

-o results.json writes everything evalkit knows about a run: the suite, target, seed, a summary (pass rate, score, flaky and errored counts, tokens, cost, p50 and p95 latency, cached runs), and every task with every run, output, grade, and reason. format: 1 versions the layout. compare reads only the summary and each task's id, status, passed, and score.