aite
Star
Fork
Watch
aite docs design protocol.md
12 kB · 151 lines
Markdown
at main

Runner and extension protocol (v1 draft) #

Transport: one websocket per run, the extension is the client, the bridge is the server. ws://127.0.0.1:<port>/ext?token=<run token>. (One bridge, and so one extension connection, per profile: a run whose script uses several profiles starts a bridge for each, with its own port and token, and each Chrome's extension connects to its own. The runner routes a request by the profile it carries; the bridge and the extension never see that key.) The bridge listens on 127.0.0.1 only and closes any connection with a wrong token.

Handshake #

ext    -> { "type": "hello",   "extVersion": "0.1.0", "chrome": "154.0.0.0", "profile": "test-qian" }
bridge -> { "type": "welcome", "runId": "2026-10-09T13-00-00Z-ab12", "protocol": 1 }

If protocol differs from what the extension supports, the extension shows an error badge and closes.

Requests and responses #

Every request from the bridge has a numeric id. The extension answers with the same id. Exactly one response per request.

{ "id": 7, "cmd": "resolve", "target": { "text": "Send" }, "timeoutMs": 5000, "tab": 123 }
  -> { "id": 7, "ok": true, "descriptor": { ... }, "matches": 1, "repaired": false, "via": "exact" }

resolve may carry "descriptor", the element as the lock file remembers it (the full shape of data-formats.md). Without it the target is resolved exactly and nothing else is tried. With it, and only when the exact target finds nothing (not-found; an ambiguous or hidden match is never repaired), the page tries the remembered fields that name an element on their own (data-testid, then id, aria-label, name, each only if it leads to exactly one visible, enabled element), then scores the visible, enabled elements (at most 2000 looked at, the best 60 by a quick comparison described in full; see locator-repair.md). Answers:

  -> found by a remembered field or by scoring:
     { "id": 7, "ok": true, "descriptor": { ...the element now... }, "matches": 1, "repaired": true, "via": "descriptor" | "repair",
       "repair": { "score": 0.902, "runnerUp": 0.31, "old": { ... }, "changed": ["text", "name"], "changes": ["text \"Send\" -> \"Send now\""] } }
  -> a candidate between 0.60 and 0.85, never acted on:
     { "id": 7, "ok": false, "reason": "drifted", "detail": { "target": {...}, "candidate": { ... }, "score": 0.744, "runnerUp": 0.1, "old": { ... }, "changed": [...], "changes": [...] } }
  -> two candidates within 0.15 of each other:
     { "id": 7, "ok": false, "reason": "ambiguous", "detail": { "target": {...}, "repair": true, "candidates": [ { "descriptor": { ... }, "score": 0.88 } ], "old": { ... } } }
  -> nothing close enough: the plain not-found, plus "repairTried": true, "bestScore", and a "note" when the page had more than 2000 elements

{ "id": 8, "cmd": "act", "action": "click", "descriptor": { ... } }
  -> { "id": 8, "ok": true }
{ "id": 9, "cmd": "verify", "kind": "text", "value": "1 MALLBUCKS", "within": "Review send", "timeoutMs": 5000 }
  -> { "id": 9, "ok": true, "found": "1 MALLBUCKS" }
{ "id": 11, "cmd": "tabs" }
  -> { "id": 11, "ok": true, "tabs": [ { "id": 123, "url": "...", "title": "...", "active": true } ] }
{ "id": 12, "cmd": "navigate", "tab": 123, "url": "https://..." }
  -> { "id": 12, "ok": true, "tab": 123, "url": "https://..." }

outline draws boxes for the screenshot and removes them again (P4-2):

{ "id": 13, "cmd": "outline", "items": [ { "xpath": "/html/body/a", "label": "possible match 0.74", "tone": "candidate" } ] }
  -> { "id": 13, "ok": true, "drawn": 1 }
{ "id": 14, "cmd": "outline", "clear": true }
  -> { "id": 14, "ok": true, "removed": 1, "remaining": 0 }

tone is wanted (blue), candidate (pink) or anything else (orange); at most 10 boxes, labels cut to 80 characters, text only. Each box is a temporary div of our own (attribute data-aite-overlay, inline styles only, pointer-events: none) added to the document element: the page's own elements, attributes and style sheets are not touched. Drawing again replaces the boxes; the page removes them itself after 15 s if nobody clears them. The runner draws only around the screenshot of a failed step (the drifted candidate, the matches it could not tell apart, or the element an action failed on), clears right after, and records a warning with code outline if drawing failed or remaining was not 0.

navigate without tab opens a new tab, remembers it as the run's tab and answers with its id; resolve, act and verify without tab go to that tab (tab-closed if there is none yet). navigate waits for the page to finish loading (timeout after timeoutMs, tab-closed if the tab goes away) and only opens http and https pages. A page command that dies because the page is navigating under it (navigation) is sent again to the new page until its own timeoutMs if it only looks at the page (resolve, verify); an act is never sent again. Tabs that were open before the extension loaded get the content script injected on demand (Chrome does not do it for them), and a tab whose origin is not in the allow list fails as internal with a message naming the allow list.

{ "id": 13, "cmd": "switchTab", "match": { "urlContains": "/pay" }, "tab": 123, "timeoutMs": 5000 }   -> { "ok": true, "tab": 124, "url": "...", "title": "..." }
{ "id": 13, "cmd": "switchTab", "match": { "new": true }, "tab": 123 }                                  -> the newest tab that opened by itself, not yet taken
{ "id": 14, "cmd": "closeTab", "tab": 124 }                                                              -> { "ok": true, "closed": 124, "tab": 123 | null, "alreadyClosed"?: true }

switchTab waits (until timeoutMs) for a matching tab to exist and finish loading, never counts the tab named by tab (the one the run is on), and brings the tab to the front. Two matching tabs is ambiguous; none in time is timeout with the open tabs' addresses. closeTab goes back along the tabs the run switched from; tab in its answer is where the run is now, or null.

Page commands (the extension asks the content script in the run's tab; all of them only look at the page, so all may be sent again, except where act says):

{ "id": 20, "cmd": "locate", "descriptor": { ... }, "timeoutMs": 5000 }
  -> { "ok": true, "x": 130, "y": 210, "rect": { "x": 100, "y": 200, "width": 60, "height": 20 }, "tag": "button", "editable": false, "kind"?: "checkbox"|"radio", "checked"?: true }
{ "id": 21, "cmd": "read", "target": { "css": ".balance" } }                                             -> { "ok": true, "text": "5 MALLBUCKS" }
{ "id": 22, "cmd": "verify", "kind": "targetText", "target": { ... }, "value": "...", "negate"?: true, "within"?: "..." }
{ "id": 23, "cmd": "wait", "kind": "visible" | "gone" | "textChanges" | "url", "target": { ... } | "contains": "..." }
{ "id": 24, "cmd": "act", "action": "select", "descriptor": { ... }, "option": "Dark blue" }

locate scrolls the element into view, requires it visible and enabled, requires the middle of it to hit the element itself (not-visible with coveredBy otherwise), and answers where to click, in viewport pixels. The runner then sends the real mouse or keys over the debugging pipe; act with click, doubleClick, type, press, check, uncheck or drag is only used as plain events when there is no pipe. verify kinds: exists, text, targetText, value, state, count, url, title, numberChange. wait network-idle is answered by the runner from Chrome's network events, not by the extension.

Failure response:

{ "id": 9, "ok": false, "reason": "mismatch", "detail": { "expected": "1 MALLBUCKS", "found": "1 MUCK" } }

Retries. An answer may carry "retries": N (a whole number, 1 or more; absent when there were none). It counts the times the same command was sent again: by the extension to a new page after the page navigated under it (resolve, verify), and by the bridge after a reconnect (only resolve, verify, locate, read, wait, tabs). The two add up. An action is never sent again, so its answer never carries retries. The runner counts them and shows them; it never hides one.

reason is one of: not-found, ambiguous, not-visible, not-enabled, timeout, mismatch, drifted, navigation, load-failed (a page the browser could not load: it showed its own error page), tab-closed, connection-lost, unsupported, pre-state, internal. (pre-state is used since P5-7.a: a precondition of the test, such as a balance, was not met before the first step ran; it is distinct from an app failure.) detail holds what the report needs (candidates, expected, found, a proposed descriptor for drifted).

Screenshots are not a bridge message #

The extension cannot take them: chrome.tabs.captureVisibleTab needs the <all_urls> or activeTab permission (specific host permissions are not enough, checked in Chrome 155), and <all_urls> would remove the extension's main safety property, that it only touches the origins in the allow list. activeTab is only granted by a user gesture. So the runner takes the picture itself, over Chrome's debugging protocol on a private pipe (--remote-debugging-pipe: no network port, usable only by the process that started Chrome). It asks the extension for the tabs list to learn which url the run's tab shows, finds the page with that url in Chrome (and its title if two tabs share a url; if that is still not enough the capture is refused, not guessed), brings it to the front and captures a PNG. A picture that cannot be taken is a warning on the step, never a failed step.

Events (extension -> bridge, no reply) #

{ "type": "event", "kind": "tab-opened",  "tab": 124, "url": "..." }
{ "type": "event", "kind": "tab-closed",  "tab": 124 }
{ "type": "event", "kind": "navigated",   "tab": 123, "url": "..." }
{ "type": "event", "kind": "console",     "tab": 123, "level": "error", "text": "..." }   # page errors, shown in the report

Keeping the MV3 service worker alive #

A Manifest V3 service worker is stopped when idle. An open websocket that sends or receives a message at least every 20 s keeps it running in current Chrome. So:

  • The bridge sends { "type": "ping" } every 15 s; the extension answers { "type": "pong" }.
  • If the worker is stopped anyway, the extension reconnects (backoff 0.5 s, 1 s, 2 s, 4 s, 8 s, then every 8 s) and sends hello with the same runId field: { "type": "hello", ..., "resume": "<runId>" }.
  • A chrome.alarms alarm every 30 s wakes the worker as a backup.
  • Close codes the bridge uses: 4401 wrong token, 4408 no hello in time, 4409 replaced by a newer connection, 4410 no answer to pings, 4426 protocol mismatch (sent by the extension). After 4401 the extension shows the error badge and keeps retrying (the launcher may have rewritten bridge.json). After 4409 another extension has taken over the bridge: the extension shows the error badge and stops retrying until the worker restarts, so two extensions sharing a token cannot fight over the bridge.

Delivery rules #

  • Requests are not retried blindly. After a reconnect the bridge re-sends only idempotent requests that had no response (resolve, verify, locate, read, wait, tabs). An act that had no response fails the step with connection-lost; the runner never clicks twice.
  • Commands run one at a time per run. The bridge does not send request N+1 before it has the response to N.
  • Message size limit 8 MB, enforced by both ends.

Security #

  • Token is 128 random bits, generated per run. The runner writes it, with the bridge address and the profile name, to bridge.json in the extension's own folder (<userDataDir>/aite-extension/, mode 600) before Chrome starts; the extension reads it as one of its own packaged files each time it connects (so a new run's token is the one used: measured in P1-6), and the runner deletes the file when the run ends. It is never in a URL the page can see. The extension ignores messages from web pages for this channel; page scripts cannot reach the service worker.
  • The extension requests host permissions only for the origins listed in the run's allow list in aite.config.json.