Action families and observation channels#

Reference for every built-in action family, its operations, payload shape and return value, plus the observation channels and the privileged/actor split. Derived from crates/environment/src/lib.rs (Environment::dispatch, Environment::observe, Environment::scene) and crates/environment/src/desktop_extensions.rs. When this file and the code disagree, the code is right.

The grant model#

EnvironmentConfig is the whole grant list:

{"actor":"alice","machines":["alice-mac"],
 "actions":["terminal.v1","pointer.v1","keyboard.v1","application.v1","browser.v1"],
 "observations":["semantic.v1","terminal.v1"],
 "action_budget":1024}

• actions — families the session may dispatch. An unknown family is rejected at environment() time, not at step() time.
• observations — channels observe() returns, and the gate on scene()/render().
• action_budget — maximum actions per step() batch (default 1024). A larger batch is refused whole, before any action runs.
• machines — every ActionEnvelope.machine must be in this list.

step() checks machines.contains(action.machine) && actions.contains(action.family) before dispatch; failure is a per-action denied outcome, not an exception. Errors are flattened to four codes (denied, not_found, invalid, action_failed) with fixed messages, so a failing action cannot leak world state through its error text. A refusal also carries a reason from a closed vocabulary — which grant is missing, or what is not there — whose message is likewise a compile-time constant. See Errors and refusals for the whole table.

Two presets exist in crates/protocol/src/lib.rs:

Built-in action families#

The seven built-in families are BUILTIN_FAMILIES in crates/environment/src/lib.rs. Every one is actor-available: possession of the family in actions is the entire permission. There are no owner-only operations inside a family — the privileged/actor split is at the handle level (see below).

terminal.v1#

The result is also latched into the session's terminal.v1 observation channel.

Two levels of failure, and they are not the same. outcome.success reports whether the action dispatched; exit_code reports whether the command worked. A command that does not exist, or a flag the shell does not implement, returns success: true with a non-zero exit_code and an explanatory stderr — exactly as a real shell would. A provider that checks only success will read every failed command as a success.

awk x  ->  success: true, {"exit_code": 1, "stderr": "awk: command not found: awk\n"}

The shell refuses rather than approximates. find . -newer foo does not silently drop the predicate; it fails with find: unsupported predicate `-newer` . sed accepts s/PATTERN/REPLACEMENT/[g] and says so when given anything else. Unsupported surface is visible at the point of use, so a task built on it fails loudly instead of producing a plausible wrong answer.

filesystem.v1#

Paths are resolved against the machine's cwd. list checks execute+read access as the machine's user and fails denied; a missing path is not_found. These are the actor's own filesystem calls, subject to simulated permissions — not the owner's unrestricted view.

http.v1#

Routed through the simulated network from the acting machine. No host socket is involved.

browser.v1#

All ops return the active page projection ({"version","title","elements","theme"}). Dispatch first forces the browser visible and, on a themed desktop, focuses or launches a browser window; per-window browser state is swapped in and out so each window keeps its own history and tabs.

The omnibox#

navigate, and Enter in a focused address bar, take a line of typed text and read it the way Chrome's omnibox does, in this order:

1. A leading ? forces a search: ?github.com searches for github.com.
2. An explicit scheme is honoured as typed. https://github.com/northstar/atlas is fetched as it stands, and file:///etc/passwd is refused with an error rather than quietly searched for — a programmatic caller that passes a bad URL hears about it.
3. Text shaped like a host becomes https:// plus that text: github.com, github.com/northstar/atlas, intranet.internal, 10.0.1.10, localhost:8080, mail.google.com/threads/3. Host-shaped means a dotted name whose last label is shaped like a TLD, an IP literal, a name with an explicit port, or a single label that the world's DNS resolves — so intranet works where intranet is a real name. foo.bar is host-shaped; foo bar is not, and neither is 3.14, v1.2 or alice@northwind.example.
4. Anything else is a search on the default engine: https://google.com/search?q=<urlencoded>.
5. A host-shaped line whose name does not resolve becomes a search after all, so a mistyped query never lands on an error page. A name that resolves but refuses the connection still shows the error page, because that host really is there.

The default engine is a per-browser template (%s stands for the urlencoded query); a world states its own in metadata.search_engine, so a world with no google.com points at the engine it has. It is part of the browser's serialised state and is written only when it is not the default, so older snapshots load unchanged.

Any other op is invalid. Note the cross-family gate: application.v1 launch of kind browser, keyboard.v1 key of Enter in a focused address bar, and pointer clicks that resolve into page content all additionally require browser.v1 in actions and are denied without it. That denial carries the reason browser_family_required, and application.v1 list marks such an application launchable: false with the same reason, so an installed-but-ungranted browser is never a bare refusal.

application.v1#

list is how an agent finds out what is on a machine without guessing ids. id is the kind launch takes; label is the name this machine's shell paints under the application's icon — the same string, from the same table, that titles its window; kind is builtin (one of the four the compositor implements), native (an application shipped with the simulator) or web (a desktop_apps alias that opens a site in the browser). installed reflects the computer's installed_apps; launchable says whether launch would succeed right now for this session, and blocked_by names the documented reason it would not — typically application_not_installed or browser_family_required. Passing {"installed": false} returns the uninstalled entries too, so "what could this world have?" is answerable as well as "what does it have?". From the shell, apps lists the same ids.

shell invokes a shell control by name, reaching the same handler a pointer reaches by hit-testing, through the same grants. The target is any shell:* id, or a window:<id>:* id, which focuses that window first. This is how an agent drives a control without locating its pixels, and how a test asserts a control's behaviour independently of where a shell happened to paint it. A target that is neither is invalid.

Shell interaction targets#

Every one of these is routed by shell_action / desktop_panel_action, and every one is reachable either from a painted control or from application.v1 shell.

File manager controls, reached as window:<id>:content:<target>: files-back, files-forward, files-up, files-root, files-home, files-reload, files-newtab, files-tab:<i>, files-closetab:<i>, files-location:<path>, files-view, files-sort:<key>, files-search, files-search-clear, files-recents, files-browse, files-starred, files-star, files-star:<i>, files-quick-access, files-gallery, files-trash, files-hidden, files-open, files-new-folder, files-new-file, files-cut, files-copy, files-paste, files-rename, files-delete, and open:<i>, which indexes the displayed row order rather than the raw listing.

Visual Studio Code controls (kind code), reached as window:<id>:content:<target>. Every one dispatches into the same command table the Command Palette, the menus and the keybindings use, and a control whose command cannot run now is painted disabled with the reason:

Keys follow VS Code, with Meta treated as Cmd, i.e. as Ctrl: Ctrl+S, Ctrl+Z/Ctrl+Y, Ctrl+X/C/V through the machine's text clipboard, Tab/Shift+Tab, Ctrl+/, Alt+Up/Down, Shift+Alt+Up/Down, Ctrl+Shift+K, Ctrl+Enter, Ctrl+G, Ctrl+F/H, F3, Ctrl+B, Ctrl+J, Ctrl+` , F5/Ctrl+F5, Alt+Z, Ctrl+, and the Ctrl+K chords (Ctrl+K Ctrl+O Open Folder, Ctrl+K Ctrl+T theme, Ctrl+K M language). A single typed character is a keystroke (brackets and quotes close and are typed over, Enter keeps the indentation); a longer keyboard.v1 type is inserted as written, the way a paste is, so its own indentation is not indented again.

FreeCAD controls (kind freecad), reached as window:<id>:content:<target>. Commands carry FreeCAD's own names; one the document cannot take now (Pocket with no solid, Fillet with no edge selected) is painted disabled with the reason, everywhere it appears.

Keys: Ctrl+N/O/S/Shift+S/I/E/Z/Y/R/A, Delete, in a file dialog Enter/Escape (answer the frontmost prompt, else OK/Cancel), Alt+ArrowLeft/Ctrl+[ back, Alt+ArrowRight/Ctrl+] forward, Alt+ArrowUp/Ctrl+ArrowUp up, Escape (drops the tool in hand, then the selection, then leaves the sketch or cancels the task), Enter (commits a field, finishes a polyline, OKs a task), arrows pan and PageUp/PageDown zoom. With no field focused, typing 0–6 turns to the standard views and a space toggles the selection's visibility, as FreeCAD's single-key shortcuts do. Typed text goes to the focused field; values take units (25, 25 mm, 1 in, 45 °, 0.5 rad).

KiCad controls (kind kicad), reached as window:<id>:content:<target>. KiCad opens one window per frame — project manager, Schematic Editor, PCB Editor, Simulator, Symbol Editor, Footprint Editor, 3D Viewer — all over the one open project. A control whose command cannot run now is painted disabled with the reason; while a dialog is open only its own controls act:

Typed letters with no dialog field focused are the frame's hotkeys (W wire, A symbol, P power, L label, Q no-connect, J junction in the schematic; R run and P probe in the simulator).

kind accepts text_editor as an alias for editor and file_manager for files. Built-in window kinds are browser, files, editor, terminal, plus the native applications (calendar, mail, messages, docs, notes, contacts, settings, calculator, clock, photos, music, maps, weather, code, freecad, kicad, the image editors paint, preview, pixelmator, gimp, pinta, sketchbook, the spreadsheets spreadsheet and excel, the SQLite client database, and the video editors clipchamp, imovie, kdenlive and videoeditor) listed by the native_apps! macro in crates/applications/src/apps/mod.rs. A world may also declare desktop_apps metadata aliases that launch a browser window at a fixed URL. Launching a kind the machine does not have installed is not_found.

Video editor controls#

The video editors are interfaces over one engine (crates/video, see video-editing.md): Clipchamp (clipchamp) on Windows 11, iMovie (imovie) on macOS and iOS, Kdenlive (kdenlive) on Ubuntu and the Android video editor (videoeditor). Launched with a folder as argument the import sheet starts there (the default is Videos on Windows and Ubuntu, Movies elsewhere); a .cwvideo project or a media file opens or imports it. Every control is window:<id>:content:video:<command>; a command the product does not have (Clipchamp's reverse, iMovie's track lock) is refused, and a control that cannot act now is painted disabled with its reason.

Keys: Space play/pause, arrows step a frame, Home/End, Delete, Ctrl+Z/Ctrl+Y (Meta+Z/Meta+Shift+Z on the Mac), Ctrl+=/Ctrl+- zoom, Ctrl+S save, Ctrl+I import, Escape closes a sheet or clears the selection; split is S in Clipchamp, Meta+B in iMovie and Shift+R in Kdenlive; J/K/L shuttle in iMovie and Kdenlive; Shift+Delete ripple-deletes and S toggles snapping in Kdenlive. With no field focused, typed letters are these shortcuts.

Music player controls#

One player (crates/applications/src/apps/music) with five faces: Apple Music on macOS and iOS, YouTube Music on Android, Media Player on Windows 11 and Rhythmbox on Ubuntu. Everything on screen comes from the media service the world backs it with (spotify.com, or music.youtube.com on Android): GET /api/catalog is the catalogue with this listener's library, likes, playlists, history, player and speakers, and every control is a real request, after which the catalogue is read again. Controls are window:<id>:content:music:<command>.

Keys: Space (or MediaPlayPause) toggles playback with no field focused, MediaTrackNext/MediaTrackPrevious skip, Enter searches or creates, Escape closes a menu, a popup or the composer.

Image editor controls#

The image editors are interfaces over one engine (crates/raster): Windows 11 Paint (paint:), macOS Preview (preview:) and Pixelmator Pro (pixelmator:), GIMP (gimp:) and Pinta (pinta:) on Ubuntu, Sketchbook (sketchbook:) on Android, and the phones' photo editors inside Photos (photos:edit:). Launched with an image path as argument, an editor opens that file (PNG or JPEG; BMP on the desktops; GIMP's layered XCF in GIMP). Every control is window:<id>:content:<prefix>:<command>; a command the product does not have (Paint's Gaussian blur, GIMP's shape tools) is refused.

Drag surfaces. canvas:<w>:<h> (the image view), slider:<param>:<width>, dial:<param>, curve:<w>:<h>, hscroll:… and vscroll:… follow the pointer: pointer.v1 down on one captures the pointer — on a phone too, before any gesture — and every move and the up are delivered to it relative to where it was painted, even outside it. A brush paints along the drag, a selection or shape spans it, a slider takes the value under the pointer. click on a surface is a press and release at one point (a dot, a fill, a colour pick). The text tool types where it was clicked; keyboard.v1 type fills it and Enter stamps the glyphs, rasterised in the platform's font by the renderer. A drag on the canvas reports the crosshair cursor. pointer.v1 move over the canvas with no button down is delivered to it too: Paint, GIMP and Pinta show the pixel under the pointer in their status bars, and the desktop editors outline the brush tip (and a clone's source) under the pointer. The Move tool shows the layer moving during the drag.

Spreadsheet controls#

spreadsheet is each platform's own spreadsheet over one engine (crates/sheet): Excel on Windows, Numbers on macOS and iOS, LibreOffice Calc on Ubuntu and Google Sheets on Android; excel is Microsoft Excel as a second application on the Mac. Launched on nothing they open on ~/Documents (Excel's Open page, Numbers' document browser, Calc's Start Center); launched with a .xlsx, .ods or .csv path they open that file. Files are real bytes on the machine: new workbooks save as XLSX, or ODS from Calc; a file keeps its own format when saved again. Every control is window:<id>:content:sheet:<command>.

Drag surfaces. sheet:grid:<row height>:<scale> is the cell area and sheet:fill:<row height>:<scale> the fill handle at the corner of the selection. A drag on the grid selects from the press to the release (the press stays the active cell); while a formula is waiting for an argument (=SUM() a drag inserts the range instead. A drag from the fill handle fills the selection in the direction dragged furthest, continuing number and date series and adjusting relative references. sheet:chartmove:<i>:<row height>:<scale> is a chart: pressing selects it and a drag moves it, landing on the cells (and offsets into them) under its new corners. sheet:chartsize:<i>:<handle>:<row height>:<scale> are the eight sizing handles of the selected chart, clockwise from the top-left (0) to the left edge (7). In Draw Border mode a drag on the grid draws the outline (or grid, or erases) of the cells it spans. While a formula is typed its references are coloured in the editor and their cells outlined in the same colours. A double click on the grid edits the active cell; on a phone a tap selects and a second tap (a double click) edits. Keys follow Excel with Meta as Ctrl: arrows (with Shift to extend, Ctrl to jump to the edge of the data), Tab, Enter, F2, Delete, Backspace, PageUp/PageDown, Ctrl+Home/End, Ctrl+A, Ctrl+C/X/V, Ctrl+Z/Y, Ctrl+B/I/U, Ctrl+D/R, Alt+= and Ctrl+S.

Database controls#

database is DB Browser for SQLite on Windows and Ubuntu and TablePlus on macOS, over the crates/sql engine; phones have none. Launched with a .db, .sqlite, .sqlite3 or .db3 path it opens that SQLite file. Changes stay in the open connection until Write Changes (TablePlus's Commit, Ctrl+S) writes the whole file; Revert Changes returns to what the file holds. Every control is window:<id>:content:db:<command>.

A double click on a grid cell edits it; Enter or Tab commits, Escape cancels, Delete sets NULL. The shell's sqlite3 works on the same files.

keyboard.v1#

Routing is by focus, in order: an open shell panel (search field), a focused registered application, a focused address bar, a visible browser, otherwise the desktop/window. Meta, Super, Meta+Space and Ctrl+Escape open the launcher on a themed desktop; Alt+Tab cycles windows.

pointer.v1#

x/y are clamped to ±32768; width/height default to the machine's screen as the actor last addressed it (the last width/height a pointer action stated), or before any to the shell's native screen (390×844 on iOS, 412×915 on Android, 1280×800 on a desktop), and are capped at 8192. That same last-addressed size, with its orientation, is what a screenshot (shell:screenshot, Recents' Screenshot) captures. Coordinates are in the same viewport you pass to scene(width, height) — hit testing runs against that scene, so the actor aims using only what it can see. cursor is one of default, pointer, text, grab, grabbing, crosshair (an image editor's canvas), ns-resize, ew-resize, nesw-resize, nwse-resize.

Behaviour worth knowing:

• Scrolling is a platform service. A pane whose content is taller than its view is published in Scene::scrolls (ScrollArea: target, window, bounds, offset, extent, horizontal, and on iOS the large title that collapses into the navigation bar once offset >= title_height). A wheel first goes to the application's own use of the wheel, then to the innermost pane under the pointer that can still move that way, then outwards. Applications with their own wheel: Visual Studio Code's editor and terminal (whole rows), spreadsheets (three rows a notch, Shift for columns, Ctrl zooms), SQLite grids (rows), KiCad canvases (zoom about the pointer; Shift pans up and down, Ctrl left and right, as KiCad's defaults), image editors' canvas (scroll, Shift across, Ctrl zooms about the pointer), FreeCAD's 3D view (zoom), terminals (lines), and the video editors' timelines (video:lanes, video:ruler, video:clip: the wheel scrolls through time, Shift or a sideways wheel too, and Ctrl/Cmd zooms about the frame under the pointer; a view put somewhere by hand stays there until the playhead moves or plays). A plain-text editor's rows are the pane:text area: the view scrolls by whole rows independently of the caret, and an edit or caret move brings the caret back into view; editor-text:<first row> always names the row painted first. A video editor's media bin is the pane:bin area. The browser's page is the pane:page area; its offset is browser.v1 scroll's y. Offsets are window state (Window::scroll) and survive snapshots.
• Some panes scroll sideways. A shelf of album covers (Apple Music's and YouTube Music's shelves, and a page Row with Style::scroll_x) is published horizontal: its offset and extent run along x. delta_x moves it, and so does delta_y with Shift held; a plain delta_y passes it by to the upright pane around it. On a page such a shelf is pane:row:<row id>, which browser.v1 scroll also takes by name.
• Scroll bars are real. On desktops each overflowing pane paints a thumb sized to the real extent, on a drag surface window:<id>:content:pane:<name>:<track>:<thumb>:<max> (hpane:…, along the bottom edge, for one that scrolls sideways): dragging the thumb scrolls, a press on the track jumps the thumb there, and a named click without a point pages forward. Phones paint no bar at rest, as they do not.
• Phones scroll under the finger. A touch that moves more than 12 px mostly vertically, starts inside the application in front and is no shell gesture (not from the status bar, the home indicator or the navigation bar) takes the pane it started on and taps nothing. Every move then scrolls it so the content stays under the finger (an application's own wheel use, a grid or a scrollback, moves a row at a time); pulled past an end, the content follows with iOS's rubber-band resistance (Scroll::stretch) and springs back on release. On up the pane flings on: the velocity is the last two finger samples over the world-clock time between them (a 60 Hz frame when the clock did not move), and the distance is the platform's deceleration (iOS 0.998/ms; Android's OverScroller spline), stopping at an end. A finger that rested 40 ms of world clock before lifting does not fling. A down/up with no move between scrolls by exactly the distance moved. A mostly sideways touch moves a shelf that scrolls sideways under it the same way.
• A phone's Back is the application's first. Android's Back button (shell:mobile-back) and iOS's navigation-bar chevron go to the application's parent screen before leaving it: Mail's message to its mailbox, a mailbox (iOS) to Mailboxes, Gmail's drawer closed, a conversation to the Messages list, a note or document to its list, a contact to Contacts.
• Text focus is one answer. A native application reports the text field that has the focus (focus.keyboard.target, role textbox); a phone paints its keyboard exactly when there is one, and on a phone keyboard.v1 type with none reaches nothing. Messages' composer and a note's or document's body take the focus when tapped on a phone, and whenever their conversation or document is open on a desktop.
• down then up is a click with press identity. down records the hit target and its bounds; up only fires if released inside those bounds. Dragging out cancels, as on a real pointer.
• Double-click is real. On a desktop theme, a single click on a desktop icon or file-manager row selects; double_click opens. On ios/android themes there is no such distinction and a tap opens immediately. double_click on a window title bar (window:<id>:drag) maximizes.
• button: 2 on down opens the context panel — except over an application drag surface that uses the secondary button (FreeCAD's 3D view pans with a right drag). button is passed to drag surfaces, so a middle drag can differ from a left one.
• Touch gestures are synthesized from down/up on mobile themes (a move of more than 70 px, mostly along one axis; a smaller one is a tap). A finger coming down on a control only presses it; the release decides whether it was a tap or a swipe.
  • iOS: down from the status bar opens Notification Center, or Control Center when it starts right of the Dynamic Island (shell:gesture:notifications / :control-center); up from the bottom edge — the home indicator — goes home (shell:gesture:home), and a swipe longer than a third of the screen opens the App Switcher (shell:gesture:overview); up anywhere puts away Notification Center or Control Center. On the home screen, right to left moves to the next page and, past the last, opens the App Library; left to right moves to the previous page and, before the first, opens Today View (panel calendar); down opens Search. Left to right in the App Library returns to the last page, right to left in Today View to the first. Left to right along the bottom edge in an application switches to the previous one. The home indicator is painted only where the device shows one: over applications, Settings and Notification Center, not on the home screen.
  • Android (three-button navigation): down from the status bar, or anywhere on the home screen, opens the notification shade, and a second pull expands it to Quick Settings; up closes the shade; up on the home screen opens the app drawer and down closes it. Sideways on the home screen walks Pixel Launcher's pages (the dots above the hotseat are shell:home-page:<n>); in Recents it walks the carousel, and past the oldest card is Clear all. Swipes that start on the 48 px navigation bar are presses of its buttons: Back (shell:mobile-back), Home (shell:home), Recents (shell:overview).
  • On both, a card swiped up in the overview / App Switcher closes that application. Tapping the space around the cards goes home.
• Window drag and resize:<edge> operations capture the pointer between down and up; while captured, move/up bypass hit testing. So do an application's drag surfaces (an image editor's canvas and sliders; see Image editor controls), and a drag that starts on one is never taken for a touch gesture.

Interaction targets#

Scene nodes carry an interaction string. Clicking one is how everything in the shell is driven, so these strings are part of the actor-facing contract.

Every shell:* target requires application.v1. An unrecognized shell:* target is invalid.

Extension families#

Environment::register_action_family adds a family; register_observation_channel adds a channel. Both reject names that collide with built-ins. Registration is an owner operation on trusted code. An extension still only runs for a session that lists its name in actions/observations, and it receives &mut Runtime, so it is responsible for its own access checks. Registered module versions are part of the snapshot compatibility boundary.

Observation channels#

observe() returns {"tick": u64, "channels": {name: value}}. Built-in channels are keyed by machine inside the channel value.

scene(width, height) returns the structured scene for the focused machine and requires semantic.v1 or pixels.v1; without either it is denied. Viewport must be 1..=16 777 216 pixels. render(width, height) rasterizes that same scene to {"width","height","rgba"} and is gated identically.

Granting semantic.v1 without pixels.v1 is a meaningful configuration only if you also refuse to call render(); the runtime does not distinguish them as gates.

Privileged (owner) versus actor#

The split is by object, not by operation name. This is what lets an evaluator grade an episode honestly: the grader reads ground truth through a handle the agent provably never had.

The last three rows — registration, runtime() and interfaces() — are Rust-only; they hand out unrestricted engine access and are deliberately absent from the Python and Wasm bindings.

The bindings enforce this structurally rather than by convention. In Python, World and Environment are separate classes and the actor object has exactly five attributes — id, step, observe, scene, render — with no snapshot, inspect, state_hash or trajectory; examples/python/smoke.py asserts that. In Rust, world.actor(&session)? hands out an ActorEnvironment<'_> borrow with the same four methods.

Consequences for evaluation:

• An actor can never read the world definition, so task-relevant facts cannot be recovered by introspecting the environment instead of using the computer.
• trajectory() and inspect() give a grader ground truth the agent has no path to.
• state_hash() over the owner handle detects whether two runs diverged at all, at semantic rather than pixel granularity.
• The action journal is tamper-evident (action_journal(), verified on import_snapshot), so a recorded episode can be replayed and checked rather than trusted. enable_replay_verification(true) additionally hashes after every step and fails replay_diverged at the first mismatching step.
• fork(snapshot) branches a checkpoint cheaply, which is how you score several continuations from one state without re-running the prefix.

Never hand a World to the agent under evaluation. Hand it the session handle.