30 kB · 299 lines

Markdown

Alex Wykoff · fix(engine): a page that did not load fails at once, ambiguous targets fail fast, visibility follows the browser · 2w ago

c13d33a0

Script format (v1 draft) #

Files: scripts/<area>/<name>.aite, UTF-8, one test per file in v1. Lines are trimmed; indentation is cosmetic (two spaces by convention). # starts a comment to end of line (not inside a string). Blank lines are ignored.

Grammar #

file     := test
test     := "test" STRING header* step+
header   := "covers" ID+                       # journey ids, e.g. J-09 J-13
          | "start" URL                         # first page to open
          | "needs" "session" NAME              # profile/account required (see PLAN.md section 9)
          | "timeout" DURATION                  # default per-command timeout (default 10s)
step     := "step" STRING command+              # the STRING is the intent and is required
command  := VERB arg* modifier*
arg      := TARGET | STRING | NUMBER | NAME
modifier := KEY ":" VALUE                       # within:"Review send"  timeout:5s  as:balance
TARGET   := KEY ":" VALUE ( KEY ":" VALUE )*    # text:"Send"   role:button name:"Send"
VALUE    := STRING | BARE                       # BARE = letters, digits, - _ . /
STRING   := '"' ( [^"\\] | '\\"' | '\\\\' | '\\n' )* '"'
DURATION := NUMBER ("ms" | "s" | "m")
ID       := letter+ "-" digit+

A file that does not match the grammar fails to parse with a message of the form file.aite:12: <what was expected>.

Targets #

A target describes an element by how a person would. The runner expands it to a full descriptor on first resolve (see data-formats.md).

Key	Matches	Notes
`text:"Send"`	element whose visible text equals (trimmed, case-insensitive)	`text~:"Sen"` for contains
`role:button name:"Send"`	ARIA role (explicit or implicit) with accessible name	preferred for buttons and links
`label:"Amount"`	form control by its label
`placeholder:"Search"`	input by placeholder
`testid:"send-btn"`	`data-testid` attribute	most stable; ask the app to add them
`css:".balance"`	CSS selector	last resort
`xpath:"//button[2]"`	XPath	last resort
`index:2`	choose the 2nd match when several	1-based; adds a warning in the report

Several keys in one target are ANDed. More than one match without index: fails as ambiguous (the report lists the candidates). It fails after about a second and a half (AMBIGUITY_GRACE_MS), not after the command's whole timeout: waiting longer cannot tell the elements apart, unless the page is still drawing, which is what the grace is for. A target that is absent or hidden still gets the whole timeout, because a slow page is a different thing.

Commands (v1) #

Command	Form	What it does
`start`	`start URL`	open the URL in the test tab (header form also allowed)
`goto`	`goto URL`	navigate the current tab
`click`	`click TARGET`	scroll into view, check visible and enabled, click
`doubleClick`	`doubleClick TARGET`
`type`	`type TARGET "text"`	focus, replace the value, fire input and change events
`press`	`press KEY`	key on the focused element: `Enter Tab Escape ArrowDown`
`select`	`select TARGET "option text"`	choose an option of a select
`check` / `uncheck`	`check TARGET`	checkbox or radio
`verify`	see below	passes when the condition holds within the timeout
`verifyNot`	same forms as verify	passes when the condition does not hold for the whole timeout
`wait`	see below	wait for a condition, then continue
`sleep`	`sleep 2s`	fixed pause; allowed but reported as a warning
`capture`	`capture NAME TARGET`	read the element's text into a variable
`set`	`set NAME "value"`	define a variable; use as `${NAME}` inside strings
`switchTab`	`switchTab url~:"/pay"` or `switchTab new`	focus another tab or window (wallet pop-up)
`closeTab`	`closeTab`	close the current tab and return to the previous one
`drag`	`drag TARGET to:TARGET`
`screenshot`	`screenshot "name"`	save a named screenshot into the run
`call`	`call NAME arg*`	run a TypeScript custom check from `scripts/checks/NAME.ts`
`run`	`run path/to/other.aite`	include another script's steps

Custom checks that ship with the runner: call session-check (is the profile signed in as the configured account), and call balance-at-least AMOUNT [LABEL] (reads the account's balance and fails with reason pre-state, not an app failure, when it is below AMOUNT; the failure names the account, the balance and the shortfall). A check reads text with read(target, {blocks: true}) when neighbouring blocks must not run together (handle and balance); without blocks the text is as the page joins it.

verify forms #

verify  TARGET                       # exists and is visible (default)
verify  TARGET  text:"1 MALLBUCKS"   # element text equals
verify  text:"1 MALLBUCKS"           # any element with this text exists and is visible (exact, case-sensitive)
verify  text-i:"1 mallbucks"         # the same, ignoring case (the whole text)
verify  text~i:"mallbucks"           # contains this text, ignoring case
verify  TARGET  text-i:"..."         # the loose forms work with a TARGET too (an element whose text is ...)
verify  TARGET  value:"alex"         # input value equals
verify  TARGET  enabled | disabled | checked
verify  TARGET  count:3
verify  TARGET  attr:href equals:"/gift/42"      # an attribute equals (a link's href: as written, or as the browser resolves it)
verify  TARGET  attr:href contains:"/gift/"      # ... or contains
verify  download name:"report.zip" size-min:1000 # a file arrived in this run's download folder, whole, named so, at least this big
verify  download name~:"report"                  # ... whose name contains this
verify  url~:"/gift/"                # current URL contains
verify  title:"Mallbucks"
verify  TARGET  number:changed-by  from:NAME  equals:-1     # numeric change against a captured value

Any verify may add within:"text or target" to restrict the search to the element containing it, and timeout:5s.

wait forms #

wait  TARGET                         # becomes visible
wait  TARGET  gone                   # disappears
wait  TARGET  text-changes           # text differs from when the wait started
wait  url~:"/claim"                  # URL contains
wait  network-idle  timeout:10s      # no requests for 500 ms

Timing and honesty rules #

Every command has a timeout (header timeout, else 10 s; per-command timeout: overrides). The runner polls every 100 ms. Page loads are the exception: start and goto wait at least 10 s however short the header timeout is (it is for elements and text, and scripts keep it short to fail fast, while how fast a page loads is another matter); a header over 10 s applies to loads too, and a timeout: on the start or goto itself is exact. A page the browser could not load (it shows its own error page: the site is down, cannot be reached from this machine, or the address is wrong) fails start or goto at once as load-failed, naming the address, instead of passing the step and failing one step later as a missing element.
Visible means what the browser shows. An element counts as visible when the browser's own computed style shows it and it has a box on the page. The hidden attribute is not read on its own: it hides an element only through the browser's style rule, and a page's CSS can override that (a control with hidden and display: flex is on the screen, so verify finds it and verifyNot fails).
A verify/wait never passes by default after the timeout. A timeout is a failure with reason: timeout.
sleep is a warning in the report, so pauses that hide a race are visible.
Retries are counted in run.json and shown in the report. A step that passed only on a retry is marked passed-after-retry.

Errors the parser must produce (examples) #

Code	Message
E001	`file.aite:3: a step needs a name: step "what you want to happen"`
E002	`file.aite:7: unknown command "clik" (did you mean "click"?)`
E003	`file.aite:9: click needs a target, for example text:"Send"`
E004	`file.aite:12: unterminated string`
E005	`file.aite:14: unknown target key "txt" (text, role, name, label, placeholder, testid, css, xpath, index)`
E006	`file.aite:2: "needs session test-qian" but no such profile is configured` (runner, not parser)
E008	(retired: the runner has every command of the table above, so nothing is refused as "not supported yet")
E010	`file.aite:1: a script must begin with: test "name"`, `only one test per file`, `a test needs at least one step`, `a step needs at least one command`, `covers must come before the first step`, `start needs an http or https URL`, `timeout needs a duration such as 10s, 500ms or 1m`, `covers needs ids like J-09`, `needs must be: needs session NAME`, `unexpected "Send"` (structure and header problems)
E011	`file.aite:6: unknown check "nope": the built-in checks are session-check, balance-at-least` (only where checks are built in, as in the extension; the runner looks `call` names up in scripts/checks when it runs)
E007	`file.aite:20: ${total} is used before it is set (capture or set it first)`

Example #

test "send 1 MALLBUCKS to a handle"
  covers  J-09 J-13
  start   https://mallbucks.malldao.xyz/
  needs   session test-qian

  step "remember the starting balance"
    capture before  css:".balance"
  step "find the recipient"
    click  text:"Send"
    type   placeholder:"Search"  "alex"
    click  text~:"@distributed.camp"
  step "review shows the right amount"
    type   label:"Amount"  "1"
    click  role:button name:"Send"
    verify text:"1 MALLBUCKS"  within:"Review send"
  step "balance goes down by one within 5 seconds"
    verify css:".balance"  number:changed-by  from:before  equals:-1  timeout:5s

The parsed tree (what `parseScript` returns) #

parseScript(text) returns { ok: true, script } or { ok: false, errors }. It never throws, whatever the text. Errors are sorted by line; each is { code, line, message } and prints as file:line: message. Line numbers are 1-based and count every line of the file, blank and comment lines included.

Script  { name, covers: string[], start?, startLine?, needsSession?, needsLine?, timeoutMs?, steps: Step[] }
Step    { name, line, commands: Command[] }
Command { verb, line, ...fields below, timeoutMs? }       # timeoutMs only when the command wrote timeout:, in ms

A header that was not written is absent from the object (not undefined, not empty). covers is [] when absent. A timeout header is timeoutMs on the script. line on a command is the line of its verb. start as a header is start/startLine on the script, not a command.

Verb	Fields
`start`, `goto`	`url`
`click`, `doubleClick`, `check`, `uncheck`	`target`
`type`	`target`, `text`
`select`	`target`, `option`
`press`	`key`
`sleep`	`ms`
`capture`	`name`, `target`
`set`	`name`, `value`
`switchTab`	`tab`: `{ kind: "url", contains }` or `{ kind: "new" }`
`closeTab`	(none)
`drag`	`target`, `to` (a target)
`screenshot`	`name`
`call`	`name`, `args: string[]` (strings, numbers and names, as written, without quotes)
`run`	`path`
`verify`, `verifyNot`	`kind` and the fields of that kind (below), `within?`
`wait`	`kind` and the fields of that kind (below), `within?`

A target is an object from key to value: text:"Send" is { text: "Send" }, text~:"Sen" is { "text~": "Sen" }, role:button name:"Send" is { role: "button", name: "Send" }. Values are strings (index:2 is { index: "2" }), with escapes already resolved. The same key twice in one target is an error (E003).

verify and verifyNot kinds (the same forms for both; verifyNot has the same tree with verb: "verifyNot"):

Written	`kind`	Fields
`verify TARGET`	`exists`	`target`
`verify text:"X"` (text is the only key)	`text`	`value`
`verify TARGET text:"X"` (TARGET has some other key)	`targetText`	`target`, `value`
`verify text-i:"X"` / `verify text~i:"X"` (also after a TARGET)	`text` / `targetText`	the same fields, and `match`: `"ignoreCase"` or `"containsIgnoreCase"` (absent for the exact `text:`)
`verify TARGET value:"X"`	`value`	`target`, `value`
`verify TARGET enabled` / `disabled` / `checked`	`state`	`target`, `state`
`verify TARGET count:3`	`count`	`target`, `count` (a number)
`verify url~:"/gift/"`	`url`	`contains`
`verify title:"X"`	`title`	`value`
`verify TARGET number:changed-by from:NAME equals:-1`	`numberChange`	`target`, `from`, `equals` (a number)
`verify TARGET attr:NAME equals:"X"` / `contains:"X"`	`attr`	`target`, `attr`, `op` (`"equals"` or `"contains"`), `value`
`verify download name:"X" size-min:N` / `name~:"X"`	`download`	`name`, `nameMatch` (`"contains"` for `name~:`, else absent), `sizeMin`. Answered by the runner, not the page

wait kinds: wait TARGET is visible (target); wait TARGET gone is gone (target); wait TARGET text-changes is textChanges (target); wait url~:"/claim" is url (contains); wait network-idle is networkIdle.

within:"..." is the string on any verify or wait. as:NAME is accepted after any command and stored as as. A sleep duration is ms; sleep 2s is { verb: "sleep", ms: 2000 }. drag A to:B writes the destination as one key and value straight after to:, for example drag text:"Card" to:text:"Done".

Variables. ${NAME} inside a string is a reference. It is an error (E007, on the line of the string) when no earlier set NAME or capture NAME in the file defines NAME, in any step before it or earlier in the same step. \${ is not special: there is no escape for it in v1.

Other errors the parser gives (E003, "needs"): a command missing what it needs (type without text, press without a key, sleep without a duration, goto without a URL, verify with nothing to check, a verify combination that is not in the table above, index: that is not a whole number of 1 or more, a duplicate key in a target). E002 suggests a known command when one is within two edits, otherwise the message has no hint.

Lines for structure errors. a script must begin with: test "name" is line 1. a test needs at least one step is the line of the test line. a step needs at least one command is the line of the empty step. only one test per file is the line of the second test. A word that is not a command, outside any step, is unexpected "WORD" on its line; a known command outside a step is VERB must be inside a step. A string that is not closed on a test or step line is E004 on that line and the test or step still counts as written, so it causes no further errors.

What the runner does today (P2-3) #

Every command of the table above, and every verify and wait form, runs. How, and what each failure means:

Real input. click, doubleClick, type, press, check, uncheck and drag are sent as real mouse and keyboard events over Chrome's private debugging pipe (docs/decisions.md D11), after the page has been asked where the element is (locate): it is scrolled into view, must be visible and enabled, and the middle of it must be the element itself and not something laid over it. An element a person could not click fails (not-visible, naming what covers it; not-enabled) and nothing is clicked. Without the debugging pipe (tests only) the same commands use plain events in the page.
type clicks the field, selects its whole content and types the text as key presses, so it replaces what was there and the page sees every key and input events; change fires when the field loses focus (press Tab). type ... "" empties the field. An element that cannot take typing (not a text field, read-only, disabled) fails not-enabled.
check / uncheck click only when the box is not already as wanted, and then confirm it is: a click that changes nothing is mismatch. A radio button can be checked, not unchecked.
select chooses the option whose text (or value) equals the given text and fires input and change. An option that is not there is not-found and lists the options.
press names: Enter Tab Escape ArrowDown ArrowUp ArrowLeft ArrowRight Backspace Delete Space Home End PageUp PageDown, or one character. It goes to the focused element.
drag presses on the middle of the first element, moves to the middle of the second in small steps with the button held, and releases. It sends mouse events, which is what pages that implement dragging with mousedown/mousemove/mouseup receive. HTML5 drag and drop (draggable, dragstart, drop) was not measured and is not claimed.
verify, all forms. text: is exact and case-sensitive (white space trimmed and collapsed): "1 MALLBUCKS" does not match "11 MALLBUCKS" or "1 Mallbucks". It waits for the condition to hold, and a timeout is a failure, never a pass: not-found / not-visible when the element is not there, ambiguous when two match, mismatch (with expected and found) when it is there and different. number:changed-by reads the first number in the element's text (thousands commas and a unit after it are fine) and compares it with the first number of the captured text. within:"TEXT" restricts the search to the box that has TEXT in it: the nearest element around the innermost one holding TEXT that holds more than TEXT (so within:"Review send" is the dialog with that heading); two such boxes is ambiguous.
verify TARGET attr:NAME equals:/contains: compares an attribute of the one visible element the target names (two matches is ambiguous). For href and src the value may be written as in the HTML (/gift/42) or as the browser resolves it (https://site/gift/42): either matches, and a mismatch shows both. An element without the attribute is a mismatch that says so.
verify download looks in the folder of this run's downloads (results/<run id>/downloads/, where Chrome is told to put every file the run downloads). It holds when a finished file has the wanted name (name~: a part of it) and is at least size-min bytes (default 1: an empty file is not a download); a file Chrome is still writing (NAME.crdownload) does not count. It waits for that like any verify, and a failure says what the folder holds (no such file; the download folder holds a.txt (4 bytes), still arriving, smaller than the 1000 bytes wanted, or no file was downloaded). It needs Chrome's debugging pipe, which every real run has. verifyNot download passes only if no such file shows up for the whole time.
verifyNot passes only if the condition does not hold at any moment of the whole timeout; it fails at once, with mismatch, when it does. A hidden element does not count as there; with several matches it fails if any of them satisfies the condition.
wait: TARGET waits until it is visible; gone until nothing visible matches; text-changes until the text differs from when the wait began; url~: until the address contains the text; network-idle until no request has been in flight for 500 ms. (network-idle stays in v1: it is answered from Chrome's network events over the debugging pipe and needs no extension permission, D14.) A wait that runs out of time is timeout, with what it saw. network-idle counts requests from Chrome's own network events, switched on at the first command that can start one (so a request already running before the first real input is not seen), and ignores event streams and pings, which are open on purpose.
capture NAME TARGET keeps the element's text (or a field's value) in NAME for ${NAME} and from:NAME.
goto URL moves the tab the run is on. start URL inside a step goes to the tab the run's start opened, or opens one. switchTab url~:"/pay" waits for a tab whose address contains the text (two matches is ambiguous, none in time is timeout listing the open tabs), brings it to the front and acts there from then on. switchTab new takes the newest tab that opened by itself since the run began (a pop-up). closeTab closes the tab the run is on and goes back to the one before; a tab that had already closed itself (a pay window) is a warning, not a failure. A command in a tab that has closed is tab-closed.
screenshot "name" saves a picture under shots/NNN-name.png; a picture that cannot be taken is a warning.
run path/to/other.aite puts that script's commands in the run at that point. The path must be a .aite file inside the project (the folder with aite.config.json); no loops, no more than 8 levels. The included script's own start, covers, needs and timeout are ignored. Variables set before the run are visible in it; ones it sets are not visible to the script that included it. The report names the included file and line.
Several profiles in one script. A script may say needs session NAME more than once: the first is the default profile (the start page and every command without on: use it), the others are extra. A command picks the profile that does it with the modifier on:NAME (goto https://app.test/claim on:test-buster, click text:"Claim" on:test-buster); on: names a profile the script declares with needs session (otherwise E003, which says to add the line), and an included script (run) may use the profiles of the script that includes it. Each profile has a Chrome of its own with its own tabs, so a profile's first page is its own goto (start opens the default's). as:NAME is not this: it names a value. A profile that fails fails only its own steps, with the usual reasons (connection-lost when its Chrome or its extension goes away); as always the first failed command stops the test and the rest are skipped. Each command in the report and in run.json says which profile did it.
call session-check [HANDLE] (scripts/checks/session-check.ts, logic in runner/sessionCheck.ts) passes when the page shows the account the profile is signed in as, and otherwise fails with the instructions to renew it (docs/runbook-sessions.md): signed out or has expired, Wrong account (naming both), or cannot find which account is signed in. A profile that names its account in aite.config.json is a real signed-in session, and a script on it must begin with this call: otherwise the runner refuses it before Chrome starts (exit 2, "the first command on that profile must be: call session-check"). In the extension the same judgement is also made for every job that names a profile with an account, once the start page is open and before any step (#70): signed out or another account fails the start step with the reason wrong-session; a page that does not say who is signed in goes on (recorded unknown in archive.json). The check is given profile and account besides args, vars, read and url; read can name the whole page with css:"body".
call NAME ARGS runs scripts/checks/NAME.ts (NAME is letters, digits, - and _; nothing else is ever loaded). The default export receives { args, vars, read(target), url() } and returns nothing or true to pass, or false / { ok: false, message, expected, found } to fail (mismatch); an AssertionError is a mismatch too; any other thrown error is internal (the check crashed); it is stopped at the command's timeout (so read waits up to half of it for its element).
The first failing command stops the test; the rest of its step and every later step are reported as skipped. The runner never runs a failed command again.
${NAME} in any string is replaced with the value of the earlier set or capture. sleep waits, and is a warning in the report and in run.json.
A test with no verify passes with the warning "this test has no verify command, so it cannot fail on what the page shows".
Retries (a page command sent again after the page navigated, or after a reconnect) are counted and shown, and a step that needed one is passed-after-retry.
aite run writes results/<run id>/run.json (see data-formats.md).
verify text-i: and verify text~i: are the explicit loose forms (ignoring case; the whole text, or a part of it). They are only for verify/verifyNot; verify text: stays exact. text: and text~: in a target (click, type, and so on) stay loose, as in the table above (decided in P2-5, D14): a person finds a button by its label whatever its casing, and only a check must be exact.
start opens a new tab for the run. Scripts inside a folder called fixtures/ get that folder served on the start page's port (or use --serve DIR).

The verify guard (P6-8) #

An AI that is asked to make a red test green can do it by deleting what the test checks. On a merge request, npm run check:verifies (CI job verify-guard) compares every changed .aite file with its old version and fails when:

a verify or verifyNot was removed, turned into a different kind (for example verify into verifyNot, or into a wait), or the whole script was deleted;
a verify now expects something else: another text, number, attribute, state or count, a looser match (text: to text-i: or text~i:), or another target;
a verify was given more time (within:, timeout: or the script's timeout header).

A verify is matched by what it checks, not by the step it sits in: renaming a step, moving a verify to another step, reordering, adding verifies and editing other commands are fine. Renamed files are followed. A new script has nothing to compare with. A person lets a deliberate change through by putting the label allow-weaker-verify on the merge request; the job then prints what was weakened and passes. Exit 3 means the check could not run (no base revision, git failed) and is red too.

Who may `call` what (P7-5) #

call NAME runs scripts/checks/NAME.ts, which may be written by an AI, so three things hold:

Only listed files run. scripts/checks/allowlist.json holds the SHA-256 of every approved check (npm run checks:allow rewrites it; a person reads git diff scripts/checks and commits it). A check that is not listed, was changed after it was listed, is a symbolic link, or sits in a folder that is a symbolic link is refused, and so is every check when the list is missing or invalid. A listed check can still import other files; the allowlist covers the file, not what it pulls in, so keep checks free of local imports other than runner/ helpers.
A change under scripts/checks/ needs a person. On a merge request, CI job checks-guard (npm run check:checks) fails when anything under scripts/checks/ is added, changed or deleted, unless the merge request carries the label checks-reviewed.
A check runs in its own process with a stripped environment. It is given PATH, LANG, LC_ALL, TZ and TMPDIR and nothing else, so AITE_GITLAB_TOKEN and the profile folder are not in its environment. It asks the runner for page text and the address over a message channel. This is not a sandbox: it runs as the same user and could still read files that user can read.

Not done yet: wrapping page text as untrusted in the repair context handed to an AI (P7-5.c) waits for aite author (P7-2), which is what hands that context to an AI.

The `posts` header (P5-7.d) #

posts on a line of its own before the first step says the test publishes something public (a post, a gift message). aite schedule counts every run of such a script against a daily cap (--post-cap N, default 10, counted per UTC day, stored in the state folder) and skips the run after it with a note in the cycle's output. A run that failed still counts, since it may have published. A cap of 0 skips every posting script. Scripts without the header are never limited by that count. A script without the header that clicks something that reads like a publish button (post, publish, tweet, skeet or share as a whole word in the target) is refused when a job runs it (#71): add posts if it does publish. The word list is a hint, so the header is still the tester's word.

The `spends` header (#69) #

spends on a line of its own before the first step says the test spends money or other value (a purchase, a transfer, a tip). It is separate from posts: a script may say either, both, or neither. It is the capability the distributed design asks for in section 7, and it is off at every layer until each one agrees:

The script says spends (an included script that says it makes the including script spend: the flag is read from the expanded script, on the central system and again on the device).
The job allows spends (budget.spendsAllowed, a whole number from 0 to 1000, signed with the rest of the job; missing is the same as 0). The central system refuses to make a job for a script that spends without it (job create --spends N, dispatch --spends N).
The person at the device switched spending on (aiteWorker.setAllowSpending(true); default off, separate from posting, and enrolling again switches it off). A device without it gives the engine no spends, and the run is refused with the test spends money or other value and this job allows no spends before any page opens.
The dispatcher holds a daily cap, per UTC day, per device and for the whole system, smaller than the post cap on purpose: 3 and 5 by default. The run after the cap is recorded as skipped with the reason, as posting is.

Running a script that spends directly with aite run is not limited by this header; it is a rule for work the central system hands out.