CI integration¶
GitHub Actions¶
name: Evals
on: [pull_request]
permissions:
contents: read
jobs:
evals:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v5
- uses: actions/setup-python@v6
with:
python-version: "3.12"
- run: pip install sic-evalkit
- run: |
mkdir -p evalkit-results
evalkit run evals.yaml \
--output evalkit-results/results.json \
--junit evalkit-results/junit.xml \
--markdown evalkit-results/summary.md \
--badge evalkit-results/badge.json
env:
OPENAI_API_KEY: ${{ secrets.OPENAI_API_KEY }}
- if: always()
run: cat evalkit-results/summary.md >> "$GITHUB_STEP_SUMMARY"
The run step writes results.json, junit.xml, summary.md, and badge.json to
evalkit-results, and the last step appends the Markdown summary to the job summary. The step
fails when the suite fails, which fails the job. Add any other evalkit run arguments, such as
--tag smoke, --fail-under, or --baseline and --threshold, to the same command.
Baselines in CI¶
A simple pattern: upload results.json as an artifact on main, download the latest one in pull
request workflows, and pass it with --baseline. The response cache makes the main-branch run cheap,
and you can cache .evalkit/ between jobs with actions/cache to skip repeat API calls.
Other CI systems¶
Run the CLI and publish the JUnit file with your system's test report feature. Install with pip:
pip install sic-evalkit
evalkit run evals.yaml --junit evalkit-junit.xml -o results.json
Badge¶
--badge badge.json writes a shields.io endpoint badge such as
{"schemaVersion": 1, "label": "evals", "message": "7/8 passed", "color": "yellow"}. Publish the
file anywhere public, such as GitHub Pages or a gist, and point
https://img.shields.io/endpoint?url=<url-of-badge.json> at it.
Results format¶
-o results.json writes everything evalkit knows about a run: the suite, target, seed, a summary
(pass rate, score, flaky and errored counts, tokens, cost, p50 and p95 latency, cached runs), and
every task with every run, output, grade, and reason. format: 1 versions the layout.
compare reads only the summary and each task's id, status, passed, and score.