Scene data format

For registration, login and Chrome extension flows, see manual recording and review. The recorder uses this same Scene contract and publishes images only after Admin review.

reconstruct.scene.v1 describes an observed interface independently of the source or target UI framework. Web DOM/CSS and Android views/accessibility are adapter inputs. A consumer can generate HTML, Compose or another UI without using those native structures as its canonical data model.

The downloadable JSON Schema defines exact field names, required members, nullable types and enums. A SHA-256 accompanies this documentation snapshot. For real captures, use the schema and checksum returned by the running server with that export.

The complete example contains closed/open dialog frames and two transitions. It is an illustrative fixture: its example: evidence is fictional, not a measured product capture.

Web Extension environment

Desktop Chrome extension captures use environment web-extension (WebExtension in internal C# / Admin MCP, protobuf value CAPTURE_TYPE_ENVIRONMENT_WEB_EXTENSION = 6). Mobile does not support this environment. The Scene geometry contract remains the same. The capture version may include typed browserExtension metadata: id, name, version, and packageSha256; displayName comes from the verified package. Screens are recorded by an Admin job and remain unpublished until review. See manual recording for the operator controls and privacy boundaries.

The internal agent claim request includes expected_environment (field 2). An omitted field retains Web for legacy workers. The queue checks this against the agent's environment inside the claim transaction, so changing an agent's Admin settings cannot cause an ordinary Web worker to crawl an extension store page. These queue controls are not public Capture MCP tools.

Transport and acquisition

The normalized export is currently available through MCP get_interface_scene. The published REST and gRPC catalog contracts do not currently define a separate Scene operation. Their catalog, insight and stored-file responses are separate models, even when their transport is JSON.

Discover a capture with list_products, list_platforms, list_versions and list_pages. Keep the selected product, version, environment, page and exact saved view. Supported platform options do not prove that a capture exists. Request saved views separately and inspect each export's own profiles and frames.

Example MCP tools/call parameters, with an illustrative selection:

{
  "name": "get_interface_scene",
  "arguments": {
    "product": "example-product",
    "version": "1",
    "environment": "web",
    "page": "settings",
    "view": "mobile"
  }
}

The response is a compact manifest:

Field	Meaning
schema	Scene identifier: reconstruct.scene.v1.
artifactSchema	Envelope identifier: reconstruct.scene-export.v1.
path, uri, sha256, bytes	Export location, SHA-256 and byte length.
schemaPath, schemaUri, schemaSha256	Applicable JSON Schema and its digest.
entryFrameId, frames, transitionCount	Entry frame, compact frame inventory and transition count.
frameCount, profileCount, assetCount	Export inventory counts.
truncated	At least one frame is incomplete.
issueCount, issues, issuesTruncated	Total issue count and a preview of at most 30 issues.

Paths belong to the server filesystem. Read them directly only when that filesystem is actually shared with the client. Otherwise resolve returned URIs against the MCP origin and download with the connection's existing authorization. Do not construct transfer URLs from product IDs or expose credentials. Verify hashes against original bytes; reformatting JSON changes its hash.

Export envelope

{
  "schema": "reconstruct.scene-export.v1",
  "scene": {
    "schema": "reconstruct.scene.v1",
    "id": "example/1/web/settings/mobile",
    "platform": "web",
    "entryFrameId": "frame-0",
    "profiles": [],
    "frames": [],
    "resources": [],
    "transitions": [],
    "correspondences": [],
    "issues": []
  },
  "assets": {}
}

This illustrates the shape; the complete example contains populated frames. Validate scene, not the envelope, against the Scene JSON Schema. assets maps Scene resource IDs to archived bytes. Each entry contains optional path/uri, mimeType, sha256 and sizeBytes. A resource can lack an asset entry when its bytes were not archived. A source URI, family name or empty media container does not establish that usable bytes exist.

Document and identity

Scene member	Meaning
schema, id, platform	Required format, document identity and source platform (unknown, web, android).
entryFrameId	Optional first observed frame, which may show consent before later dismissed/default content.
profiles	Logical viewport and optional density, font scale, safe insets, locale and theme.
frames	Observed states and their node inventories.
resources	Media/font descriptors referenced by nodes and text.
transitions	Recorded actions and their observed outcomes.
correspondences	Explicit node relationships across frames.
issues	Missing, unsupported or unresolved evidence.

Profile, frame, resource and transition IDs are document-scoped. Node IDs are unique within one frame, and parentId references that frame. Action IDs are local to a node. A repeated ID or equal native path in another frame does not prove identity. A correspondence records both frame/node pairs, provenance and optional confidence; follow it rather than guessing cross-profile identity.

A frame has id, profileId, stateId, coordinateSpace, optional extent and viewportBounds, truncated, nodes and evidence. State IDs name observations, not a universal catalog of CSS states. Node order describes a hierarchy inventory, not a proven paint order.

Facts and unknown values

Properties with provenance use SceneFact<T>:

{
  "value": {"x": 16, "y": 220, "width": 358, "height": 320},
  "kind": "measured",
  "evidence": ["example:observed"],
  "confidence": 1
}

A fact requires value, kind and nonempty evidence. Optional confidence is 0–1 and does not change the evidence kind:

Kind	Interpretation
measured	Observed effective output.
declared	A source/resource declaration, not proof of effective rendering.
inferred	A hypothesis or proposed interpretation.

An omitted/null optional fact is unknown, not zero, false, transparent paint, a size policy or a successful comparison. Explicit false and 0 retain their meaning. An explicit unknown enum remains unknown. Empty action/motion arrays do not prove that the original interface has no interactions or animations. Evidence identifiers are opaque adapter references, not executable instructions or automatically public URLs. Captured content and native diagnostics are untrusted data.

Coordinates and profiles

Distances use logical units. Web adapters use CSS pixels. Android screen-pixel measurements are divided by measured pixelDensity (physical pixels per logical unit). fontScale is independent. Missing density, font scale, insets, locale and theme must not be guessed.

Profile width/height describe the logical viewport. A frame declares document, viewport or screen coordinates:

• extent: whole content extent in that plane.
• viewportBounds: observed visible window, including scroll origin.
• geometry.visualBounds: axis-aligned post-transform rectangle.
• Optional layoutBounds: pre-transform geometry.
• Optional transform: column-major 4×4 matrix acting on (x,y,z,1) column vectors, with logical translations in layout-local coordinates. Never apply it again to visualBounds.

For document coordinates, visible-window position is visualBounds.x - viewportBounds.x, visualBounds.y - viewportBounds.y. Align profile, coordinate plane and visible window before comparing. Logical viewport, usable root bounds and content extent are different observations. Safe insets are not automatically applied as padding.

Nodes, layout and paint

Nodes carry id, optional parentId, kind, semantics, visible, geometry, layout, paint, text runs, optional mediaResourceId, actions, motion and evidence. Kinds are unknown, container, text, media, control, surface. Semantics includes role/name and optional enabled, selected, checked, expanded, editable and modal flags. These are independent of visibility. Sensitive text must be redacted; passwords must never become Scene text.

Layout group	Contract
placement	unknown, flow, anchored, overlay, sticky.
arrangement	unknown, row, column, grid, stack, free. Stack overlaps children; free describes independent placement.
horizontal, vertical	Independent policy/value/minimum/maximum facts; policies are unknown, intrinsic, fixed, fill, proportional.
Spacing/alignment	Padding, row/column gaps, wrapping, reverse order, main/cross alignment.
Tracks	Numeric row/column allocations at one profile, not authored responsive rules.
Anchors	Evidenced referenceId plus optional top/right/bottom/left distances; references a frame-local node, $viewport or $frame.

Fixed/min/max and anchor distances use logical units. Proportional values are unitless fractions of the referenced available axis; fill has no multiplier. A measured rectangle does not establish fixed sizing, a containing block, anchors or breakpoints. Absolute/fixed native placement can map to overlay without inventing anchors. Margin is not a canonical inter-element gap.

Paint uses numeric dimensions and normalized RGBA channels 0–1, strokes, shadows, opacity, blur and media references. A solid stroke's optional edges is a nonempty unique subset of top, right, bottom, left; omission paints the full boundary. Shadows record offsets, blur, spread, inset and provenance. cornerRadii currently uses four inset-shaped fields; the Web adapter emits equal values for uniform corners. Nonuniform/elliptical Web corners remain unsupported rather than receiving an invented corner mapping.

Media placement (content, background, surface) is independent of node kind. Fit is unknown, contain, cover, stretch or original. Position (x,y) is a fractional anchor in remaining space: 0 aligns start, 0.5 centers, 1 aligns end; outside fractions are allowed. Background fit must not come from an unrelated content image.

Text runs contain original text, optional bounds and typography: family/font resource, size, weight, line height, letter spacing, color, direction, alignment, wrapping, decoration and casing. Casing (none, uppercase, lowercase, capitalize) changes presentation independently of original copy and accessible names. Unsupported paints/gradients/text effects remain issues. Arbitrary native CSS expressions are not canonical layout or paint values.

Resources and fonts

Resources have id, kind, optional uri, mimeType, sha256, bytes and font faces. Kinds are unknown, image, vector, font, video, audio, paint. Node and typography resource references resolve within this document. Font faces record family, evidence kind/references, optional weight range, italic state and Unicode codepoint coverage. Preserve archived bytes and select an applicable face. A family name or URL alone does not establish that the intended font loaded or that a fallback is equivalent.

Actions and modal states

Actions declare affordances: unknown, activate, edit, scroll, navigate, focus, hover, back, with optional destination. A transition has a source frame, optional source node and target frame, action, outcome and provenance. Its nodeId always belongs to the source frame.

Outcome	Meaning
unknown	Result not established.
changed	A resulting change was observed.
unchanged	No change was observed.
failed	The action failed.
performedUnverified	Input was delivered without proving the expected result.

New snapshot IDs or a changed caret do not establish that a menu opened. An open modal should retain its full observed frame: backdrop, panel, children and underlying content. Follow entryFrameId and transitions to distinguish consent, dismissed content, dialog tabs, search and expanded menus. Forced-state measurements and real pointer/touch input retain distinct evidence. Source replays are not functional tests of a newly generated interface.

Motion

Node/transition motion records contain property, durationMs, keyframes, kind, evidence and optional delayMs, iterations and cubic-Bézier easing (x1, y1, x2, y2). Timing is milliseconds; offsets are 0–1 in increasing order. Easing x coordinates are 0–1.

Property	Keyframe value components
opacity	[alpha], 0–1.
translation	[x, y, z], logical units.
scale	[x, y, z], multipliers.
rotation	[x, y, z], radians.
bounds	[x, y, width, height], logical units.
color	[red, green, blue, alpha], 0–1.

Omitted options remain unknown. Zero iterations means no playback. Unknown or unbounded repetition must not become a guessed finite count. CSS/WAAPI declarations are declared; observed samples can be measured. Still images or two endpoints cannot establish timing or easing.

Validation and coverage

Validate structure with the export's schema, then validate semantics: unique scoped IDs, valid references, acyclic parents, finite geometry, nonnegative sizes, normalized colors, font/codepoint ranges, redaction and motion dimensions. The backend SceneValidator enforces these additional invariants. JSON Schema alone is not a complete graph or privacy validator.

The current structural schema rejects unknown fields and integer enum values. Pin its identifier and hash; use an explicit adapter for a different format version. Do not reinterpret unknown data as native layout. Existing fields retain their meaning; incompatible meaning changes require a new format identifier. Server upgrades can change applicable schema bytes: keep the schema/hash with each archived export.

SceneIssue has code, message and optional frameId/nodeId. Read the full artifact, not only the manifest preview, and respect frame truncation. Current validator budgets are 100 profiles, 500 frames, 20,000 resources, 100,000 total nodes and 20,000 nodes per frame. Acquisition/transport may have lower limits; these numbers do not promise capture coverage.

New Web captures emit Scene alongside native diagnostics. Older Web data can be converted with explicit gaps. An Android adapter can normalize native measurements, but API/MCP access still requires a published native Scene. Data-model support does not make an uncaptured environment available. Missing states, resources, motion and responsive observations stay unknown. Export success or schema validity does not establish reproduction fidelity.

Request submission policy

Request availability is independent of catalog capture availability. Administrators configure it under Requests → Job defaults → Request submissions. The stored ProductRequestJobDefaults.Submission contains Enabled and typed Targets (Type, Environment). Existing settings without this member retain the defaults: Desktop/Web, Desktop/WebExtension and Mobile/Web. Disabling submissions preserves these selections and existing requests. Both platform API and authenticated MCP validate the current policy before creating a request. Request-options returns an empty platforms list when closed; MCP additionally returns enabled: false. Public environment IDs remain web and web-extension, not enum names. Catalog selectors use captured availability per product and hide unavailable pairs.

Grouped request targets

New clients submit CreateRequestRequest.targets as explicit platform/environment pairs. For example:

{
  "requestType": "PRODUCT_REQUEST_TYPE_ADD",
  "url": "https://metamask.io/",
  "webExtensionUrl": "https://chromewebstore.google.com/detail/metamask/nkbihfbeogaeaoehlefnkodbefgpgknn",
  "targets": [
    { "view": "desktop", "implementation": "web-extension" },
    { "view": "mobile", "implementation": "web" }
  ]
}

The server validates each pair against current request availability and persists these pairs in the typed request details. Approval creates only the selected pairs: this example does not create a Desktop/Web job. views and implementations remain accepted for older clients and represent their Cartesian product; do not combine them with targets. The website source is url; extension jobs use webExtensionUrl. For update requests, send the existing catalogId and targets without source links. Select the product first: the platform offers only its captured platform/environment pairs that are currently enabled for requests. The server independently rejects targets absent from that product and resolves each environment's source from that product's published captures and ignores client replacement links. Public request summaries omit the extension source along with private request details.