Testing Modes

One engine, several ways to point it at your app. Each mode changes which actions Jevitate prefers — not how it verifies them.

Goal-directed testing

Give Jevitate a natural-language goal — "Create an account and verify the email", "Add a product to the cart and reach checkout". It continuously evaluates available actions relative to the goal and decides whether the goal was reached from observable application state — not because Jev returned DONE. It explores alternate paths when the obvious one is blocked, and changes strategy when repeated actions stop altering state.

jevitate explore --url http://localhost:3000 \
  --goal "Add a product to the cart and reach checkout" \
  --success urlIncludes:/checkout --real

Every --success check must hold, decided by code: URL, visibility, text, values, counts, network (requestMade, responseStatus), persistence (reloadThen), computed style and layout. The result's missionOutcome is the verdict and goalOutcome the goal's own ending.

http: end-state checks (0.15.0)

A write can have an effect that is not on the page: a password reset sends an email. When a dev outbox exposes sent mail as JSON on another origin, --success 'http:GET <url>|<assertion>[|<assertion>...]' (on explore, explore-author-journey and MCP success strings) proves it with a polled GET.

--success 'http:GET http://localhost:8025/api/v1/search?query=to:${setup.email}|jsonPath=$.messages[*].Subject|includes=Reset your password'

• Paths use $, .key, [n], ["key"] and [*]; with [*] the assertion holds when any selected value satisfies it. The URL runs to the first | (write a literal pipe as %7C).
• GET only, to an origin in the run's allowlist, with no credentials and no redirects followed (a 3xx is the answer). The body is capped at 1 MiB and never saved. An origin outside the allowlist is refused before anything opens (E_HTTP_CHECK_OFF_ALLOWLIST, exit 64).
• ${name} (a Journey param, or ${setup.x} from the fixtures) may appear only after the URL's origin and is stored as a template. A secret param or setup output is refused (E_HTTP_CHECK_SECRET_REF).
• A transport error is unevaluable, never mutation proof. Record --fixtures that empty the outbox, or an earlier email can make the check pass (lint http-check-without-reset). It satisfies no-persistence-check. See Journeys.

text= matches an element's whole text; textContains= matches part of it, case-insensitively, including content below the fold. When the checks live on different pages (connect payments, then see Pricing show it), --success-when each counts every page check once it went from not holding to holding at some settled step, each on its own page and in any order. The run stops as soon as all have held. Request path globs take {a,b} alternation, for example requestMade:POST /api/{Reschedule,Cancel}Appointment.

A run that loops ends. When the last 8 steps used at most two actions and landed on at most two page states, with no write request between them (a click ping-pong, a disclosure opened and shut, a scroll back and forth), it ends no-progress and names the loop. A page state includes its visible text and scroll position, so real change still counts as progress. The reason names the latest failed or refused action, such as type "Your answer" rejected, not an older disabled control.

A chat reply wait is bounded per message: the total wait for one reply, across every wait, is capped by --reply-ceiling-ms, and then the run ends naming the missing reply. A page whose controls are all hidden by a progress overlay is a job in progress: it is waited out within --job-wait-ms, then the run ends no-progress naming the overlay. The repeat guard ignores analytics and read-marker requests, and lets a same-labelled control fire again once its own section of the page has changed.

The typed-value guards follow what the goal or the field asks for. A value is refused as a repeat only when it was already submitted into the same live field, and never when the field's label, dialog or page text asks for it ("Type REINSTATE to confirm"). The goal-echo guard counts only the goal's instruction words: quoted text, exactly: literals, parenthesised lists and text after "a short description of", "describing", "saying" or "about" are value material. A copied instruction is still refused.

A message-named field is typed with send only when the page shows a conversation: a transcript (role=log, a live region with messages, message items) or a Send button next to it. Otherwise it is a form field. Time in an explicit wait is not render time, and a run still waiting on a reply ends no reply within Ns.

Find-out goals

Leave out --success and ask a question instead. The run ends with an answer, and code accepts it only if every claim traces back to text the run actually saw on a page. A find-out goal is read-only unless the goal asks for a change (or you pass --allow-writes). Even when it asks for a change, a goal with no --success never performs a destructive write ("Remove" → a delete request) without --allow-writes or --allow-destructive.

jevitate explore --url http://localhost:3000/contacts \
  --goal "find out how many contacts are overdue and report the count" --real

Open-ended find-outs: minimum effort, depth and partial reports

A survey-style goal ("use this tool's main features and report what works and every error") has no single fact to find. --min-actions <n> and --min-distinct-states <n> set the minimum exploration before the model may conclude: until both are met, code defers the model's report, its blocked and, when no --success check decides the run, its done, and steers the run to unvisited tabs, detail views and forms. A grounded answer reported early is kept. The minimum never ends a run and never runs forever: the budgets still stop it, a model that insists (3 deferrals in a row) gets its ending, and a minimum above the budget is capped at it with a warning.

• Default. With neither flag, a minimum applies only to an open-ended find-out (no --success check, a goal that asks for a broad survey, decided by code from its wording): 12 actions and 5 distinct page states, each at most half the budget. A narrow question concludes as soon as its answer is grounded.
• depth. Every goal result records depth: distinct states and pages, actions, decisions, forms submitted and the minimum that applied (met). The summary prints a DEPTH line.
• partialReport. A find-out that ends without a grounded answer returns partialReport: per page visited, its own text (re-checked by the grounding code), its controls and what was tried there, printed as PARTIAL lines. Nothing in it is the model's own wording.

jevitate explore --url https://app.example.test/ --max-actions 40 \
  --goal "use this tool's main features and report what works and every error you see" \
  --min-actions 15 --min-distinct-states 6

Feature testing

Specify a capability instead of a sequence: "Test checkout", "Test team permissions". Jevitate discovers the relevant UI paths dynamically, exercises multiple valid routes, explores the boundary states around the feature, and records which states and transitions were actually hit. Scope it to routes with repeatable --route globs.

jevitate explore --url http://localhost:3000 \
  --feature checkout --route "/cart/**" --route "/checkout/**"

Exploratory testing

No predefined destination. Jevitate maintains a set of visited states and prefers actions likely to reach unseen ones, discovering undocumented paths while detecting dead ends and loops. The objective flips:

maximize(new_states + new_transitions + new_behaviors)

# rather than
minimize(actions_to_goal)

jevitate explore --url http://localhost:3000/settings --strategy exploratory --real
jevitate explore --url http://localhost:3000/settings --strategy coverage --scope app --real

coverage is the breadth-first frontier and exploratory its novelty-first variant. Both stay on the start URL's route by default; --route <glob> or --scope app widens them. The run emits a coverage summary alongside one Recording per path; a discovered defect (an HTTP 5xx, an uncaught exception, horizontal overflow or vertically clipped text on a narrow viewport, a violated invariant) exits non-zero so it gates CI.

0.7.0 behaviour changes (goal runs)

• A 5xx write is unknown, not rejected. A write that answered 5xx, or failed without any response, may have been committed by the server, so its control is not clicked again unless the page offers a retry or shows an error. Only a 4xx is treated as a rejected input that may be retried.
• New text counts as progress. When an action's only effect is new text, the model is told what appeared (redacted like other page evidence) and the step counts as progress instead of "the page did not change". Text that only changes its digits, like a counter, is ignored.
• A same-labelled button that sends a different request is no longer refused. When another control in the same section, form, dialog or card has been used since the button's write, it may be clicked once more, and the next click is judged by the request it sends. A re-click with nothing used in between is still refused, as are paid or destructive controls and writes whose outcome is unknown.
• Dialog-scoped targets. On a page with an open dialog, controls below the fold that a fixed backdrop would cover are left out of the inventory, so a role=dialog without aria-modal no longer ends a run with 0 actions.
• A rejected report keeps its real reason. When the first answer was rejected and the retry found nothing, the run names the claim that could not be grounded and why.
• Nameless controls under --deny or --paid. An icon-only control named by its title, aria-labelledby or a nested image's alt shows that name in the control list, so the patterns match it; a control with no accessible name is not clicked when those patterns are declared.

0.8.0 behaviour changes (goal runs)

• The goal and the defects are separate verdicts. A defect (an HTTP 5xx, a server-log error, a violated invariant) no longer replaces a goal run's goalOutcome. Defects are counted in defectOutcome, and a goal that was not reached carries a goalReason. Exit codes are unchanged: a goal that was not reached and found no defects still exits 1; defectOutcome.status: "none" tells it apart. See the outcome table.
• Safety refusals name their rule. A refusal ends with the rule and the text it matched, for example [rule builtin:may-cost-money, matched "Generate"], so a tool that compares refusal reason strings exactly needs updating. --allow-control <regex> exempts one named benign control from the soft "may cost money" heuristic for that run, and nothing else (see Safety).
• An expired session ends the run. Before a run that starts from a session (--storage-state, --persona, queued missions), explore checks that it is still signed in. An expired one ends the run inconclusive with failure.kind: "auth-expired" and the persona's name, instead of exploring the login page. --auth-check off|auto|urlExcludes:<text>|selector:<css> sets the rule; see Sweeps & Personas.
• A toast countdown is not a hang. Toast libraries render the auto-close countdown as an indeterminate role="progressbar" and pause it while the window is unfocused, which is always the case in headless runs. An indeterminate progressbar inside a toast, snackbar or notification container, or labelled a timer or countdown, is no longer a busy indicator, so it no longer ends a run as a false ui-no-progress hang. A "Loading" progressbar in a status region still is one. A ui-no-progress hang now records the stuck indicator's nearest landmark (dialog, header, form, …) as elementContainer.
• A quiet log source is a working oracle. A --log-source that attached but read zero lines no longer makes a completed mission inconclusive; it is recorded in serverLogs.quietSources. Only a source that failed to attach or errored degrades the run. --log-quiet-ok is still accepted but no longer changes anything.
• Environment faults are not defects. Log lines that match --log-defect are classified by a per-project .jevitate/log-classes.json plus built-in rules. A placeholder or invalid API key, an unconfigured provider or a degraded health check is an environment fault: it is listed under environmentFaults and does not make the run defects-found. expected-validation lines go to expectedValidation and never fail the run. An invalid classes file is refused before any browser opens.

A job whose writes succeeded is done (0.13.0)

In goal and usability missions, a done that the page judgment rejects is accepted when all of these hold:

• code observed the job's own actions make at least one write;
• every write succeeded (2xx, none blocked; read RPCs, background, third-party and bookkeeping requests are excluded);
• the advisory scope judgment, asked in the same judgment call and at the threshold the sign-in and save rules use, says those writes are the whole goal.

The outcome carries verifiedBy: "succeeded-writes" and the note the job's writes succeeded; the page shows no change (repeat data?), which is also written to the transcript. There is no extra model call. A failed, blocked, unanswered or unlisted write, no write at all, or a scope judgment below the threshold (or unavailable) keeps the rejection, so an early done halfway through a multi-step job is still refused. A declared success check still decides when present. When the writes all succeeded but the scope judgment will not vouch for them, the rejection now says so: …; the job's writes succeeded, but they are not judged to be the whole goal (p=…).

Scope regions (0.14.0)

A single-page app that renders a multi-step sub-flow in place (same URL, no aria-modal) puts the surrounding list, its filters and the app shell into the coverage denominator, so a run that attacked the sub-flow thoroughly can still end insufficient-coverage. A scope region names the page element under test, and the threshold then applies to the controls inside it. Declare the region; don't lower --min-control-coverage.

jevitate explore --url https://app.example.test/items --strategy adversarial \
  --scope-region testId=import-view --scope-region 'role=region;name=Import'

• --scope-region <descriptor> is repeatable and uses the --success descriptor syntax (testId=import-view, role=region;name=Import); a control counts if it is inside any of them. It applies to adversarial, coverage and exploratory only. Any other strategy, a bad descriptor, or --scope-region beside an anchor that declares its own region is refused with E_EXPLORE_ARGS (exit 64).
• A Journey anchor can carry the region; a run from it uses that one and --scope-region cannot override it. Over MCP it is run_exploration.scopeRegion (a list); a suite item takes scopeRegion too.
• Only controls inside the region are enqueued or hunted for coverage; the rest stay reachable but are tried last and never count. Without a region nothing changes.
• Result fields are in the report schema. A region that never matches ends inconclusive (stop scope-unreachable), never clean.

Adversarial testing

Adversarial mode intentionally searches for unexpected behavior. It prioritizes:

• Unusual but executable state transitions and ordering violations
• Repeated and rapid interactions
• Navigation during pending asynchronous operations
• Boundary input values and contradictory UI actions
• Alternate paths through multi-step workflows

All of it stays bounded by your configurable safety policy, and the stop decision comes from a trusted hard-signal oracle — never Jev's own claim.

jevitate explore --url http://localhost:3000/settings --strategy adversarial --fake-ai

Adversarial plans its misuse in code, so --fake-ai is enough: it only replaces the advisory calls. An adversarial run that exercised too little of the form (below --min-control-coverage) is inconclusive, never clean.

• Markup injection. Boundary submits include inert canaries with a random token (an <i> element and an attribute break, never a script or event handler). If the DOM renders one as markup, after submit or after a reload, it is a markup-injection defect, stored or reflected. Text that stays escaped is not a defect.
• Identity switches. An action that changes the signed-in user (a "Continue as demo" shortcut) is listed in identityChanges; invariants are never judged across it, and the run returns to the original identity in a fresh session, or ends inconclusive (identity-changed).

0.7.0 behaviour changes (adversarial runs):

• Off-origin misuse writes are blocked. A fetch/XHR or native form post that a misuse step fires to an origin outside --allow is aborted in the browser and listed in the result's blockedWrites with its origin and how to allow it (add the origin to --allow if it is the app's, or pass --allow-write "<origin>/<path glob>"). It is never reported as a defect of the app; reads and subresources from other origins load as before. If the app refreshes a sign-in token on another origin (an identity provider) during the run, add that origin to --allow. Safety states exactly what --allow does and does not block.
• A slow save is inconclusive. A misuse step waits up to 5 s for the writes it started; if one is still pending, that step's require / always invariants are inconclusive (never a violation, never a pass) and the report names the writes it waited on. Once the save is answered, a rejected save is no longer reported as a violation because the previous "Saved" was still showing, and a page that says "Saved" over an HTTP 500 is reported as an invariant violation that verify-fix and regression capture can replay.
• A modal form is filled before a submit probe. The other empty text fields get valid values, so a double-submit, boundary-submit, edit-cancel-save or act-while-pending probe reaches the server instead of ending insufficient-coverage.
• A page's own cancelled read is not a defect. A superseded type-ahead or search read (including a read RPC sent as POST) and a document navigation replaced by another no longer produce failed-request. An aborted write and every genuine network failure still do.

Regression verification

Reproducible failures become deterministic artifacts: a minimized Recording that reproduces the smallest reliable trigger, asserts correct behavior (not the presence of the bug), and replays in CI without Jev. See Regression & CI.

Usability review

The usability strategy runs a usability review: findings that ask whether the user got the job done and where the experience gets in the way. Each finding is a claim code verified, from an opt-in guard probe (--probe-guards), your product facts (--product) or friction the run hit; Jev only categorizes and grades. It stays advisory: it never sets a non-zero exit or gates a run. Calibrate it with --app-class (e.g. consumer, admin).

jevitate explore --url http://localhost:3000 \
  --strategy usability --goal "check out" --app-class consumer --real

A usability run ignores reads its own navigation cancelled (net::ERR_ABORTED, or a gRPC-web/Connect request that answered and was then aborted), so they no longer produce "an error interrupted the task"; requests that answered 4xx/5xx, or failed without a navigation, are still reported.

It also runs offline over a saved Recording with jevitate ux <recording> --app-class. See Usability Review for the full reference, or MCP & CLI for the command surface.

How many modes, exactly?

The pluggable strategy layer:

Options every mode shares

• Watch or record it: --headed, --slow-mo, --record-video, --screenshots; --evidence-video for a captioned clip per defect (Demos & Environments).
• Emulate a device: --viewport 375x812 or --device "iPhone 13"; the size is recorded, so replays use it too.
• Repeat and vote: --repeat <n> runs it N times and reports findings seen in fewer than --min-agreement runs as flaky.
• Compare roles: --persona name=storageState (repeatable) runs it once per session and diffs requests, statuses and controls for RBAC candidates.
• Declare rules: --invariants <file>, checked around every action; --log-source correlates backend logs, matched to the exact request by trace or correlation id; --log-triage keeps only the lines related to each defect (Signal triage).
• Start inside a Journey: --from-journey <id> --at-step <n|anchor> replays a promoted Journey's prefix and starts the mission on the live page (every strategy but --feature; --fixtures, --before and --after restore state; Journeys).
• See what each action changed: --action-deltas (opt-in, not --feature) records a redacted before/after page delta and a code verdict per action.
• Browser extensions: --extension <dir> loads an unpacked extension; its chrome-extension://<id> pages are navigable (Targeting Real Apps).

Which flag applies to which mode: Flags Reference.

Input strategies

TYPE_TEXT delegates to a configurable generator, and input values are chosen by field semantics — Jevitate never assumes every field should get arbitrary fuzz. Reusable strategies include normal, empty, boundary, long, unicode, whitespace, invalid format, duplicate, and previously used. A model-invented sign-up email or username is unique per run, so a repeated sign-up never collides with an earlier account, and a goal run's --type-fixture types a file's exact text into a matching field.

Loop prevention

Jevitate tracks visited states, state/action pairs, repeated transitions, consecutive no-ops, and time spent per state — with configurable limits on max steps, duration, repeated states, repeated actions, and model/network cost.