Repository profile

A repository profile is a short, checked list of the places in a codebase where a mistake hurts most — payments, auth, migrations, public contracts, the module everything imports — with the checks a reviewer must perform when a PR touches one. ReviewStage builds it once per repository and then feeds it into every review of that repository.

skills · profiles · acme/widgets

Without a profile the agent reads each PR cold. With one, a Standard or Deep review that touches app/payments/** is told, in so many words, to verify the callers, contracts, migrations and tests behind that path and to say what it checked.

What it does to a review

When a profile exists for the repository being reviewed, run-review.sh:

• Merges the profile’s risk_paths into the risk-area banners — the same label:pattern rules as RISK_PATHS. An operator rule wins when both define a label.
• Appends a “Critical paths for this repository” section to Standard and Deep prompts (never Quick). It lists only the critical_paths whose glob matches a file in the PR’s diff — at most 12, each why cut to 200 characters — with their checks, followed by the repository’s review_rules and do_not_flag list. The agent is instructed to explicitly verify each matched path is not broken and to set critical_path on any finding that concerns one.
• Surfaces it on the PR page. A finding with critical_path carries a small critical path badge. Learnings rows keep the field, so Insights can show one more number: the kept rate of findings on critical paths.

What is gathered — no model involved

bin/profile-repo.sh <owner/name> first collects deterministic signals from the base clone with plain git, in well under a second on a small repository:

Signal	How
File tree	git ls-files, folded into directories to depth 3 (file counts per directory, root files), capped at 400 entries
Languages and manifests	Extension histogram of code files; root/one-level manifests (package.json, composer.json, pyproject.toml, go.mod, Dockerfile, …)
CODEOWNERS	The first 60 lines of CODEOWNERS, .github/CODEOWNERS or docs/CODEOWNERS
CI configuration	Names of .github/workflows/*, .circleci/config.yml, .gitlab-ci.yml, Jenkinsfile, …
Churn	Top 40 files by commits touching them in the last 12 months (git log --since=12.months --name-only)
In-degree	Top 40 files by how many other files import them, for the dominant language — relative imports for JS/TS, dotted modules for Python, class-name stems for PHP/Java/Kotlin/C#/Rust, best effort
Critical-looking directories	Any directory whose name is one of auth, payment(s), billing, migration(s), schema, api, public, webhook(s), crypto, permission(s), admin, security, session, … with its file count

bin/profile-repo.sh <owner/name> --signals-only prints exactly this JSON and exits without calling Claude. It is what the tests use, and the quickest way to see what the model would see.

Measured on the ReviewStage repository itself (161 tracked files, 118 commits in the last 12 months, TypeScript-dominant): the signals stage took 120 ms inside Python and about 0.3 s wall-clock for the whole --signals-only run.

What the model sees

One claude -p call — Sonnet by default, RS_MODEL overrides — receives the skills/repo-profile skill, the signals JSON above, and a strict output contract:

{
  "summary": "2-4 sentences",
  "critical_paths": [{ "path_glob": "app/payments/**", "why": "…", "checks": ["…"] }],
  "risk_paths": [{ "label": "payments", "pattern": "app/payments/" }],
  "review_rules": ["…"],
  "do_not_flag": ["…"]
}

The model runs in the base clone with read-only tools (Read, Glob, Grep) so it can confirm a path before naming it, and nothing else. Its reply is the JSON.

Every path_glob from a profiling run is validated against the tree. A glob that matches no tracked file is dropped and logged (dropped hallucinated path_glob … in the job log; the Skills page shows the dropped globs under the profile). Labels are restricted to [A-Za-z0-9_-], and a risk pattern containing , or : is discarded because those are the rule separators.

A glob is matched the way a person reads it: * stops at a / and only ** spans directories. fnmatch’s * crosses separators, so src/*.ts used to match all four thousand files in a tree and every review was told the critical path was touched. A glob that matches more than half the repository is now rejected outright — that is “the repository”, not a critical path.

Three limits on that sentence are worth knowing:

• Validation needs a base clone, and says when it did not have one. A profile generated by bin/profile-repo.sh always has one, so its globs are always checked. A profile edited in the dashboard is checked only when the base clone exists and git ls-files succeeds; when it does not — a repository accepted through REPO_ALLOW_ORG that has never been reviewed, a clone still in flight, a wedged checkout — the tree check is skipped and the profile records validated: false with the reason. The card says “Paths were not validated — there is no clone of this repository” rather than presenting unchecked globs as ground truth. They are validated the next time the profile is saved with a clone present.
• Every list is bounded now. risk_paths, review_rules and do_not_flag are capped at 20 entries each, checks at 8 per path, and critical_paths at 24 stored — a profile used to accept fifty and quietly carry them. The cap is applied at validation, not silently at render time, and the number dropped is recorded. At most 12 matched paths reach any one review prompt, and that cap now keeps the paths the PR hits hardest and says how many it did not list, rather than truncating in the order the model happened to emit them.
• A summary-only profile is accepted. The profile is refused only when the summary, the critical paths and the review rules are all empty. A reply that produced nothing but two sentences of summary — every glob hallucinated and dropped — is saved, and the Skills page shows the dropped globs. Check the dropped list after a run rather than assuming a saved profile is a useful one.

An edit that empties a section that was not empty is a different case, and is refused unless you confirm it. The markdown round trip used to match headings by prefix, so renaming or deleting a heading yielded an empty list and saved cleanly — deleting the risk-paths section made every risk rule disappear from future reviews with no warning. Headings are matched against the format’s own set, unrecognised ones are reported back to you rather than swallowed, and the refusal carries per-section counts.

Schema version, staleness and earlier versions

profile.json carries a version field (currently 1) and is shape-checked when it is read. A hand-edited or future-version file is surfaced as invalid — “schema version … is newer than this build understands” — instead of being read as a critical-paths section that is silently empty for ever.

Staleness is computed, not just stored. The head the profile was generated against used to be recorded, returned, typed, and never compared to anything. The card now reports:

• whether the profile is stale at all;
• how many commits behind the repository’s current head it is;
• how many of its critical paths no longer match any tracked file — which marks a profile stale even when the head has not moved.

Earlier versions are readable and restorable. Each save renames the previous profile.json to profile.<ts>.json beside it; the card counts them and they can be opened and restored. Nothing prunes them, so the count only grows — one small JSON file per edit.

Cost and duration

• The signals stage is free and fast (see the measurement above).
• The model stage is one Sonnet call. Its exact tokens and duration depend on the size of the tree and how many files the model chooses to peek at; the Skills page shows the real model, token count and cost of the last run once it has completed. As an estimate, not a measurement: expect roughly one to three minutes and tens of thousands of input tokens for a mid-sized repository. The job’s own timeout is 15 minutes, and it shares the box-wide review lock so it never runs beside a review.

Where it lives

$ROOT/profiles/<owner>__<name>/
  profile.json          the machine copy reviews read (schema version + generation metadata)
  profile.md            the human copy — edited from the dashboard, parsed back into JSON
  profile.<ts>.json     every earlier version, kept when a new one is written
  signals.json          exactly what the model was shown
  status · pid · .lock · usage.json · agent.log · run.log

Running and editing it from the dashboard

The Skills page has a Repository profile section with one card per configured repository:

• Status — never run · running, with the current phase (fetching the repository → gathering signals → asking the model → validating paths) and a Stop button · or the last run’s date, model and token count, and who edited it last, plus the staleness read above.
• Degraded signals are named. A git call that times out or is OOM-killed no longer takes the run down or, worse, implies the repository has no churn: _git never raises, the churn scan streams a year of history through a counter instead of buffering it, and a degraded list travels with the signals so the card and the prompt can say which signal was skipped.
• Profile this repo / Re-profile this repo — runs on your connected Claude account, exactly like a review; the button is disabled until you connect one in Integrations.
• Counts — critical paths · risk paths · review rules · do-not-flag entries, and how many earlier versions exist.
• The editor — the same Preview / Edit markdown editor used for findings. Save parses the markdown back into the profile (headings ## Summary, ## Critical paths with one ### glob per path, Why: and - check: lines, ## Risk paths as - label: pattern, ## Review rules, ## Do not flag), validates every path against the base clone’s tree, keeps the previous version, and records who edited it. The next review of that repository picks it up.
• Re-profile automatically when the file tree changes materially — admin only; see below.

The API behind it: GET /api/profile?repo=, POST /api/profile/run, POST /api/profile/stop, PUT /api/profile (body md or json to save, or auto_profile to flip the checkbox). Run, stop and save carry the signed profile token from the GET; run needs a connected Claude account.

Automatic re-profiling

With the checkbox on, pr-watch.sh checks that repository at most once a day: it refreshes the base clone, sorts git ls-files, and compares it to the list recorded at the last profile. If at least max(5, 2 %) of paths were added or removed, it asks the server (POST /api/profile/auto, signed with the server secret) to rebuild the profile. The server runs it as the admin, on the admin’s connected Claude account; if the admin has none, it logs auto-profile skipped … admin has no connected Claude account and does nothing. The poller never handles a Claude token.

The setting is stored in settings.json as "auto_profile": {"<owner>__<name>": true} — see Configuration.

Testing it without a model

• bin/profile-repo.sh <owner/name> --signals-only — the deterministic stage, JSON on stdout.
• python3 -m unittest discover -s bin -p 'test_*.py' — bin/test_rs_profile.py covers glob validation (hallucinated globs dropped, directory globs match files beneath), risk-rule merging, prompt assembly (cap of 12, why truncation, Quick gets nothing), the markdown round-trip, versioning, and the signals stage on a throwaway git repository with a canned model reply.
• The Playwright suite renders the Skills section from an offline fixture profile and round-trips an edit through the editor.