How checklists work
What a checklist is, which ones run on a pull request, the order they run in, and scripts against the model.
A checklist checks one thing in a pull request, such as security, tests or known vulnerabilities. A review automation is an ordered list of checklists, which starts from its template and which you can change in the automation, under What it looks for. The checklist catalog has a page for each.
Which checklists run
Each checklist has paths, globs on the changed files. It runs when a changed file, after the skip globs, matches one of them, and is skipped otherwise, so it costs nothing. ** matches any change.
A skipped checklist shows ⬜ on the summary card, and its check run is skipped with the title "No matching files changed". When no checklist runs, no model is called, unless the project has merge gates.
How each one checks
| How it checks | What happens |
|---|---|
| Script | A script built into the action finds the problems. No model call. |
| Script, then the model | A script gathers facts first, such as the packages a lockfile changed or the tests that changed with each file, and the model judges them. |
| Model | The model reads the change with the checklist's step. |
A script reads the diff, the pull request's description, the guide files and files in the checkout. It never runs the repo's code. Three scripts call the network, and all three ask the Axon Shaft API: Known vulnerabilities and Licences send it public package names and versions, which it looks up on osv.dev and the package registries (see Package lookups), and GitHub Code Quality findings sends nothing.
A script's findings are posted like the model's, inline and in the summary, and set its check run. A blocking or should-fix finding from a script raises the review's verdict. When only script checklists run, no model is called.
Each fact-gathering script runs once, however many checklists use it, and its facts are cut at about 4,000 tokens. When it fails, its facts say why and the checklist still runs.
The order
The model gets the steps in this order, numbered from 1:
- the steps that always run first;
- the step of each checklist that runs, in the automation's order;
- the project's own checks;
- the automation's own instructions, and the steps that always run last.
The summary card's chips and the check runs follow the same order.
Your own rules and checks
In the automation, under What it looks for:
- Each checklist takes up to 500 characters of your own rules, added to its step on this project.
- Your own checks, up to 20, each with a title, what to look for and the files it applies to. None means any file. They run in the model, after the checklists, and post check runs named
Axon Shaft / own-1,Axon Shaft / own-2and so on. - An automation lists up to 40 checklists.
What the docs pages leave out
A checklist's built-in rules (what it looks at, what counts as a finding, what it never flags, and what's blocking or should fix) are the prompt the model gets at run time. Neither the docs nor the dashboard show them. A checklist's page shows its summary, the files it runs on, how it checks and its check run name.
Action versions
Checklists need action 1.6 or later. A workflow pinned to an older version runs only the Code review's seven checks, and skips the rest. A checklist whose script this version of the action doesn't have is skipped, and its check run says so.
Last updated on