Regression & CI

Every finding carries its own reproduction: the Recording of the steps that led to it, the step to replay up to, and a stable fingerprint. Jevitate replays it to confirm the bug, to verify the fix, and to keep it fixed in CI.

explore ──► detect (evidence) ──► reproduce ──► commit a regression ──► verify the fix
          (code-decided oracles) (verify-fix)   (regression capture)    (verify-fix, regression run)

Is it fixed? `verify-fix`

jevitate verify-fix --result .jevitate/logs/<date>/adversarial-<stamp>.result.json --fingerprint <fp>
# 0 fixed (absent on every replay) · 1 still reproduces · 2 inconclusive · 4 intermittent · 64 unusable input

jevitate verify-fix --result <run>.result.json --fingerprint <fp> --record-video   # before/after clips

A single clean replay is not evidence of a fix: an intermittent signal can simply not fire once. verify-fix replays the defect's repro --replays times (default 3), each in a fresh session. Only absence on every replay that reached the defect's step is fixed; a signal that fires on some replays is intermittent, never fixed. It works for every finding kind: HTTP 5xx, uncaught exceptions, console errors, failed requests, hangs, invariant violations and backend-log defects. A finding found at a mobile viewport replays at that viewport.

A finding from a --from-journey run replays through its Journey prefix, in every session, as do regression capture and regression run. Give a secret param again with --param; a prefix that no longer replays is inconclusive (journey-stale), never fixed.
A finding recorded with a browser extension replays only with the same build (--extension); a different one is refused (exit 64).
--action-deltas records what each replayed step changed, as evidence.

Replaying under a corrected allow list (0.11.0)

verify-fix keeps the allow list stored with the run. Pass --allow <origin...> or --env <name> (that environment's current allow list in .jevitate/environments.json) to replay under a corrected one, for example to prove a localhost to 127.0.0.1 fix. Both together are refused, and the report records the override. Over MCP, verify_fix takes env (the result carries allowlistOverride) but refuses allow: an MCP caller never names origins, so --allow is CLI only.

Commit it: `regression capture` and `regression run`

jevitate regression capture --from <run>.json --result <run>.result.json --fingerprint <fp> --id saved-means-stored
jevitate regression run saved-means-stored              # 1 reproduces · 0 fixed · 2 inconclusive
jevitate regression run saved-means-stored --env staging # replay on another environment

regression capture writes <id>.recording.json and <id>.meta.json to the repo's .jevitate/regressions/. It first replays the failure --attempts times (default 3); a failure that doesn't reproduce every time is labelled flaky and not committed. An existing id is refused (64) unless you pass --force. What it can commit as the oracle:

A failed goal success check: appended as the final assertion, then minimized with delta debugging to the steps the failure still needs.
An app-caused failed action, or a failed network check (requestMade / responseStatus).
A declared invariant violation, re-checked on replay up to the step where it fired.
A Recording whose final step's expect fails, with no result at all.

Hard signals go to the ledger. A fingerprint naming an HTTP 5xx, a hang or a server-log defect is refused by regression capture with a pointer to ledger add / verify-fix: its oracle is the signal itself. To commit one as a regression, declare the rule it breaks as an invariant (the demo does this).

Keep it verifiable: the ledger

jevitate ledger add .jevitate/logs/<date>/feature-<stamp>.result.json <fp> --ticket JEV-123
# → .jevitate/regressions/ledger/<fp>.json  (commit it with the fix)
jevitate verify-fix <fp>          # one finding, from the ledger (no --result needed)
jevitate ledger verify            # every entry; --ticket JEV-123 or <fp>... to narrow it

Run output under .jevitate/logs/ is pruned after 14 days. The ledger keeps what a replay needs, keyed by fingerprint, in the committed regressions store: never a storage state, page text, screenshots or usage. ledger verify exits with the worst verdict; nothing to verify is inconclusive (2), never a false FIXED.

Evidence: clips and screenshots per defect

jevitate explore --url http://localhost:3000/profile --goal "save the profile" --fake-ai --evidence-video
# CLIP  .jevitate/logs/<date>/explore-<stamp>.evidence/<fp>/clip.webm
# SHOT  .../before-step-4.png  .../at-step-4.png

--evidence-video replays each defect's minimal repro (the one verify-fix replays) in a fresh session with captions. The failing step is marked with the actual signal, e.g. Save → server returned 500 (PUT /api/profile), and whether this replay saw it fire again.
Two screenshots, just before the failing step and at it (the element highlighted), sit beside the clip. The result gets defects[].evidence; the summary prints CLIP and SHOT lines.
The issue draft gets a "Repro clip and screenshots" section, and report and check link them: JUnit attachment properties and [[ATTACHMENT|…]], SARIF attachments and relatedLocations.
At most 5 defects per run get a clip, hard ones first. In check, an item that records video gets evidence by default.
verify-fix --record-video adds the pair: evidence.before (the run's clip) and evidence.after (a captioned replay now, ending on the verdict).
Registered secrets are masked in the pixels, fail closed: a capture whose mask can't be proven is skipped and the reason recorded (details). GitHub's API can't upload media, so drafts and filed issues link to the files: publish them as CI artifacts.

Gate CI: `jevitate check`

jevitate check --suite ci/jevitate-suite.json --real
jevitate check --suite ci/jevitate-suite.json --baseline last --changed-routes '/settings/**'
jevitate report --target http://localhost:3000     # one deduped defect list across modes and runs
jevitate diff <runA> <runB>                        # new / resolved / still-present / flaky / not-rerun
jevitate baseline tag release-0.2 <run>            # name a baseline for check/report --baseline

check --suite runs promoted Journeys, invariant sweeps, goals, missions and verifyFix re-checks against one or more targets, within a total action, time and spend budget.

Hard failures gate: a failed Journey assertion, an invariant violation, a failed success check, a re-check that still reproduces or is intermittent, a hard-signal defect or hang.
Advisory findings never gate (usability findings, Jev's flags) unless the suite sets "gateAdvisory": true. --fake-ai never gates a goal on the fake judge.
Fail closed: an item that crashed, was inconclusive or was refused is an error, and so is going over budget.
A mission item can start inside a Journey (fromJourney, atStep, params, env, baseUrl); atStep: "all" expands into one item per step. For many anchored missions with one report, use jevitate campaign run (Journeys).
--baseline <run|tag|last>: only findings not in the baseline gate. --changed-routes runs only the Journeys and goals that touch those routes.
Suite items take the demo options too: headed, slowMo, recordVideo, evidenceVideo, screenshots, and env / baseUrl on a Journey item.

Exit	Meaning
`0`	pass
`1`	at least one gating finding: fail the build
`2`	no gating finding, but an item errored or the budget ran out: a broken check, not a pass
`64`	the suite, its preflight, the targets file or the AI setup was refused; nothing ran

Outputs go under --out (default jevitate-check/): results/, junit.xml, jevitate.sarif (SARIF 2.1.0), check.json and report.md. The suite schema is in docs/ci.md.

Per-run unique suite params (0.16.0)

A Journey item's param may be { "unique": "inv-{run}@example.test" }. The template must contain {run}; each suite run substitutes a fresh token per entry, passes it to the run and records it in the result (secret params redacted), so repeat runs do not collide on data the app keeps. The suite keeps the template, so its identity does not change between runs. Fixture restore steps can also bind outputs and use them as ${restore.<name>} to delete what a Journey created; they are cleared before every restore pass, so each regression or check replay looks the resource up again.

Locator health and approvals in CI (0.10.0)

Locator health. Every Journey item reports its locator health: JUnit warnings per fix and SARIF warnings (rule jevitate/locator-health/brittle-locator, de-duplicated by element), never a failure. --max-brittle-steps <n> is the opt-in gate: a Journey item with more than n brittle steps gets a hard locator-health finding (exit 1) whose reproduction is jevitate locator-health --journey <id>. With the gate, a malformed .jevitate/project.json refuses the check before anything runs (E_CHECK_SUITE, exit 64). Over MCP, run_check takes maxBrittleSteps.
Verified pull-request approvals. --require-approvals --allow-channels pr-review re-verifies every pr-review approval through GitHub; one that does not verify is an unverified violation. Give the check step GITHUB_TOKEN and contents: read, pull-requests: read. See what human approval guarantees.
Machine baselines. A Journey item that ran clean records its journeyHash, baseline and locatorHealth in check.json (Report Schema).
Publishing to Journeeze. After the check, jevitate publish journeeze --json uploads the catalog bundle with those baselines. Set JOURNEEZE_UPLOAD_KEY as a CI secret (Catalog). Exit 0 imported, 1 refused, 64 a setup problem, 2 the upload could not finish.

Replay finds the recorded element exactly

A replay never clicks a guess. It uses a stable anchor captured at record time when there is one (a test id, or a non-generated, document-unique id or name); otherwise it matches the recorded role and accessible name, label or text exactly, with the recorded index among equals. A missing or ambiguous target fails the step, and verify-fix reports that as inconclusive, never fixed.

Stamping the app version (0.11.0)

jevitate check --app-version <label> (or a valid JEVITATE_APP_VERSION) records the app's release label next to targetBuild in check.json and in the SARIF run properties. It is exported to Journeeze as check.appVersion. Cold starts after a deploy are reported as advisory slowness.coldStarts notes and never change the exit code.

What a pass proves

A passing regression is behavioral evidence, not a proof of total correctness: it shows that the captured failure no longer reproduces under the encoded conditions. Failing to find more bugs never proves none exist.