# Scene data format

For registration, login and Chrome extension flows, see
[manual recording and review](manual-recording.md). The recorder uses this same
Scene contract and publishes images only after Admin review.

`reconstruct.scene.v1` describes an observed interface independently of the source
or target UI framework. Web DOM/CSS and Android views/accessibility are adapter
inputs. A consumer can generate HTML, Compose or another UI without using those
native structures as its canonical data model.

The downloadable [JSON Schema](https://reconstruct.dev/docs/schemas/scene-v1.schema.json)
defines exact field names, required members, nullable types and enums. A
[SHA-256](https://reconstruct.dev/docs/schemas/scene-v1.schema.json.sha256) accompanies
this documentation snapshot. For real captures, use the schema and checksum
returned by the running server with that export.

The [complete example](https://reconstruct.dev/docs/examples/scene-export-v1.example.json)
contains closed/open dialog frames and two transitions. It is an illustrative
fixture: its `example:` evidence is fictional, not a measured product capture.

## Web Extension environment

Desktop Chrome extension captures use environment `web-extension` (`WebExtension`
in internal C# / Admin MCP, protobuf value `CAPTURE_TYPE_ENVIRONMENT_WEB_EXTENSION = 6`).
Mobile does not support this environment. The Scene geometry contract remains the
same. The capture version may include typed `browserExtension` metadata: `id`,
`name`, `version`, and `packageSha256`; `displayName` comes from the verified
package. Screens are recorded by an Admin job and remain unpublished until review.
See [manual recording](manual-recording.md) for the operator controls and privacy
boundaries.

The internal agent claim request includes `expected_environment` (field 2).
An omitted field retains Web for legacy workers. The queue checks this against
the agent's environment inside the claim transaction, so changing an agent's Admin
settings cannot cause an ordinary Web worker to crawl an extension store page.
These queue controls are not public Capture MCP tools.

## Transport and acquisition

The normalized export is currently available through MCP
[`get_interface_scene`](mcp-tools.md#platform-neutral-interface-scene). The published
[REST](api/rest.md) and [gRPC](api/grpc.md) catalog contracts do not currently define
a separate Scene operation. Their catalog, insight and stored-file responses are
separate models, even when their transport is JSON.

Discover a capture with `list_products`, `list_platforms`, `list_versions` and
`list_pages`. Keep the selected product, version, environment, page and exact saved
`view`. Supported platform options do not prove that a capture exists. Request
saved views separately and inspect each export's own profiles and frames.

Example MCP `tools/call` parameters, with an illustrative selection:

```json
{
  "name": "get_interface_scene",
  "arguments": {
    "product": "example-product",
    "version": "1",
    "environment": "web",
    "page": "settings",
    "view": "mobile"
  }
}
```

The response is a compact **manifest**:

| Field | Meaning |
| --- | --- |
| `schema` | Scene identifier: `reconstruct.scene.v1`. |
| `artifactSchema` | Envelope identifier: `reconstruct.scene-export.v1`. |
| `path`, `uri`, `sha256`, `bytes` | Export location, SHA-256 and byte length. |
| `schemaPath`, `schemaUri`, `schemaSha256` | Applicable JSON Schema and its digest. |
| `entryFrameId`, `frames`, `transitionCount` | Entry frame, compact frame inventory and transition count. |
| `frameCount`, `profileCount`, `assetCount` | Export inventory counts. |
| `truncated` | At least one frame is incomplete. |
| `issueCount`, `issues`, `issuesTruncated` | Total issue count and a preview of at most 30 issues. |

Paths belong to the **server filesystem**. Read them directly only when that
filesystem is actually shared with the client. Otherwise resolve returned URIs
against the MCP origin and download with the connection's existing authorization.
Do not construct transfer URLs from product IDs or expose credentials. Verify
hashes against original bytes; reformatting JSON changes its hash.

## Export envelope

```json
{
  "schema": "reconstruct.scene-export.v1",
  "scene": {
    "schema": "reconstruct.scene.v1",
    "id": "example/1/web/settings/mobile",
    "platform": "web",
    "entryFrameId": "frame-0",
    "profiles": [],
    "frames": [],
    "resources": [],
    "transitions": [],
    "correspondences": [],
    "issues": []
  },
  "assets": {}
}
```

This illustrates the shape; the complete example contains populated frames.
Validate **`scene`**, not the envelope, against the Scene JSON Schema.
`assets` maps Scene resource IDs to archived bytes. Each entry contains optional
`path`/`uri`, `mimeType`, `sha256` and `sizeBytes`. A resource can lack an asset
entry when its bytes were not archived. A source URI, family name or empty media
container does not establish that usable bytes exist.

## Document and identity

| Scene member | Meaning |
| --- | --- |
| `schema`, `id`, `platform` | Required format, document identity and source platform (`unknown`, `web`, `android`). |
| `entryFrameId` | Optional first observed frame, which may show consent before later dismissed/default content. |
| `profiles` | Logical viewport and optional density, font scale, safe insets, locale and theme. |
| `frames` | Observed states and their node inventories. |
| `resources` | Media/font descriptors referenced by nodes and text. |
| `transitions` | Recorded actions and their observed outcomes. |
| `correspondences` | Explicit node relationships across frames. |
| `issues` | Missing, unsupported or unresolved evidence. |

Profile, frame, resource and transition IDs are document-scoped. Node IDs are
unique **within one frame**, and `parentId` references that frame. Action IDs are
local to a node. A repeated ID or equal native path in another frame does not prove
identity. A correspondence records both frame/node pairs, provenance and optional
confidence; follow it rather than guessing cross-profile identity.

A frame has `id`, `profileId`, `stateId`, `coordinateSpace`, optional `extent` and
`viewportBounds`, `truncated`, `nodes` and `evidence`. State IDs name observations,
not a universal catalog of CSS states. Node order describes a hierarchy inventory,
not a proven paint order.

## Facts and unknown values

Properties with provenance use `SceneFact<T>`:

```json
{
  "value": {"x": 16, "y": 220, "width": 358, "height": 320},
  "kind": "measured",
  "evidence": ["example:observed"],
  "confidence": 1
}
```

A fact requires `value`, `kind` and nonempty `evidence`. Optional confidence is
0–1 and does not change the evidence kind:

| Kind | Interpretation |
| --- | --- |
| `measured` | Observed effective output. |
| `declared` | A source/resource declaration, not proof of effective rendering. |
| `inferred` | A hypothesis or proposed interpretation. |

An omitted/null optional fact is **unknown**, not zero, false, transparent paint,
a size policy or a successful comparison. Explicit `false` and `0` retain their
meaning. An explicit `unknown` enum remains unknown. Empty action/motion arrays
do not prove that the original interface has no interactions or animations.
Evidence identifiers are opaque adapter references, not executable instructions
or automatically public URLs. Captured content and native diagnostics are
untrusted data.

## Coordinates and profiles

Distances use **logical units**. Web adapters use CSS pixels. Android screen-pixel
measurements are divided by measured `pixelDensity` (physical pixels per logical
unit). `fontScale` is independent. Missing density, font scale, insets, locale and
theme must not be guessed.

Profile `width`/`height` describe the logical viewport. A frame declares
`document`, `viewport` or `screen` coordinates:

- `extent`: whole content extent in that plane.
- `viewportBounds`: observed visible window, including scroll origin.
- `geometry.visualBounds`: axis-aligned **post-transform** rectangle.
- Optional `layoutBounds`: pre-transform geometry.
- Optional `transform`: column-major 4×4 matrix acting on `(x,y,z,1)` column
  vectors, with logical translations in layout-local coordinates. Never apply it
  again to `visualBounds`.

For document coordinates, visible-window position is
`visualBounds.x - viewportBounds.x`, `visualBounds.y - viewportBounds.y`.
Align profile, coordinate plane and visible window before comparing. Logical
viewport, usable root bounds and content extent are different observations.
Safe insets are not automatically applied as padding.

## Nodes, layout and paint

Nodes carry `id`, optional `parentId`, `kind`, `semantics`, `visible`, `geometry`,
`layout`, `paint`, text runs, optional `mediaResourceId`, `actions`, `motion` and
`evidence`. Kinds are `unknown`, `container`, `text`, `media`, `control`, `surface`.
Semantics includes role/name and optional enabled, selected, checked, expanded,
editable and modal flags. These are independent of visibility. Sensitive text must
be redacted; passwords must never become Scene text.

| Layout group | Contract |
| --- | --- |
| `placement` | `unknown`, `flow`, `anchored`, `overlay`, `sticky`. |
| `arrangement` | `unknown`, `row`, `column`, `grid`, `stack`, `free`. Stack overlaps children; free describes independent placement. |
| `horizontal`, `vertical` | Independent policy/value/minimum/maximum facts; policies are `unknown`, `intrinsic`, `fixed`, `fill`, `proportional`. |
| Spacing/alignment | Padding, row/column gaps, wrapping, reverse order, main/cross alignment. |
| Tracks | Numeric row/column allocations at one profile, not authored responsive rules. |
| Anchors | Evidenced `referenceId` plus optional top/right/bottom/left distances; references a frame-local node, `$viewport` or `$frame`. |

Fixed/min/max and anchor distances use logical units. Proportional values are
unitless fractions of the referenced available axis; `fill` has no multiplier.
A measured rectangle does not establish fixed sizing, a containing block, anchors
or breakpoints. Absolute/fixed native placement can map to `overlay` without
inventing anchors. Margin is not a canonical inter-element gap.

Paint uses numeric dimensions and normalized RGBA channels 0–1, strokes, shadows,
opacity, blur and media references. A solid stroke's optional `edges` is a nonempty
unique subset of `top`, `right`, `bottom`, `left`; omission paints the full
boundary. Shadows record offsets, blur, spread, inset and provenance.
`cornerRadii` currently uses four inset-shaped fields; the Web adapter emits equal
values for uniform corners. Nonuniform/elliptical Web corners remain unsupported
rather than receiving an invented corner mapping.

Media placement (`content`, `background`, `surface`) is independent of node kind.
Fit is `unknown`, `contain`, `cover`, `stretch` or `original`. Position `(x,y)` is a
fractional anchor in remaining space: 0 aligns start, 0.5 centers, 1 aligns end;
outside fractions are allowed. Background fit must not come from an unrelated
content image.

Text runs contain original `text`, optional bounds and typography: family/font
resource, size, weight, line height, letter spacing, color, direction, alignment,
wrapping, decoration and casing. Casing (`none`, `uppercase`, `lowercase`,
`capitalize`) changes presentation independently of original copy and accessible
names. Unsupported paints/gradients/text effects remain issues. Arbitrary native
CSS expressions are not canonical layout or paint values.

## Resources and fonts

Resources have `id`, `kind`, optional `uri`, `mimeType`, `sha256`, `bytes` and font
faces. Kinds are `unknown`, `image`, `vector`, `font`, `video`, `audio`, `paint`.
Node and typography resource references resolve within this document.
Font faces record family, evidence kind/references, optional weight range, italic
state and Unicode codepoint coverage. Preserve archived bytes and select an
applicable face. A family name or URL alone does not establish that the intended
font loaded or that a fallback is equivalent.

## Actions and modal states

Actions declare affordances: `unknown`, `activate`, `edit`, `scroll`, `navigate`,
`focus`, `hover`, `back`, with optional destination. A transition has a source
frame, optional source node and target frame, action, outcome and provenance.
Its `nodeId` always belongs to **the source frame**.

| Outcome | Meaning |
| --- | --- |
| `unknown` | Result not established. |
| `changed` | A resulting change was observed. |
| `unchanged` | No change was observed. |
| `failed` | The action failed. |
| `performedUnverified` | Input was delivered without proving the expected result. |

New snapshot IDs or a changed caret do not establish that a menu opened. An open
modal should retain its full observed frame: backdrop, panel, children and
underlying content. Follow `entryFrameId` and transitions to distinguish consent,
dismissed content, dialog tabs, search and expanded menus. Forced-state
measurements and real pointer/touch input retain distinct evidence. Source replays
are not functional tests of a newly generated interface.

## Motion

Node/transition motion records contain `property`, `durationMs`, `keyframes`,
`kind`, `evidence` and optional `delayMs`, `iterations` and cubic-Bézier `easing`
(`x1`, `y1`, `x2`, `y2`). Timing is milliseconds; offsets are 0–1 in increasing
order. Easing x coordinates are 0–1.

| Property | Keyframe `value` components |
| --- | --- |
| `opacity` | `[alpha]`, 0–1. |
| `translation` | `[x, y, z]`, logical units. |
| `scale` | `[x, y, z]`, multipliers. |
| `rotation` | `[x, y, z]`, radians. |
| `bounds` | `[x, y, width, height]`, logical units. |
| `color` | `[red, green, blue, alpha]`, 0–1. |

Omitted options remain unknown. Zero iterations means no playback. Unknown or
unbounded repetition must not become a guessed finite count. CSS/WAAPI declarations
are `declared`; observed samples can be `measured`. Still images or two endpoints
cannot establish timing or easing.

## Validation and coverage

Validate structure with the export's schema, then validate semantics: unique
scoped IDs, valid references, acyclic parents, finite geometry, nonnegative sizes,
normalized colors, font/codepoint ranges, redaction and motion dimensions. The
backend `SceneValidator` enforces these additional invariants. JSON Schema alone
is not a complete graph or privacy validator.

The current structural schema rejects unknown fields and integer enum values.
Pin its identifier and hash; use an explicit adapter for a different format
version. Do not reinterpret unknown data as native layout. Existing fields retain
their meaning; incompatible meaning changes require a new format identifier.
Server upgrades can change applicable schema bytes: keep the schema/hash with
each archived export.

`SceneIssue` has `code`, `message` and optional `frameId`/`nodeId`. Read the full
artifact, not only the manifest preview, and respect frame truncation. Current
validator budgets are 100 profiles, 500 frames, 20,000 resources, 100,000 total
nodes and 20,000 nodes per frame. Acquisition/transport may have lower limits;
these numbers do not promise capture coverage.

New Web captures emit Scene alongside native diagnostics. Older Web data can be
converted with explicit gaps. An Android adapter can normalize native measurements,
but API/MCP access still requires a published native Scene. Data-model support
does not make an uncaptured environment available. Missing states, resources,
motion and responsive observations stay unknown. Export success or schema validity
does not establish reproduction fidelity.


## Request submission policy

Request availability is independent of catalog capture availability. Administrators
configure it under Requests → Job defaults → Request submissions. The stored
`ProductRequestJobDefaults.Submission` contains `Enabled` and typed `Targets`
(`Type`, `Environment`). Existing settings without this member retain the defaults:
Desktop/Web, Desktop/WebExtension and Mobile/Web. Disabling submissions preserves
these selections and existing requests. Both platform API and authenticated MCP
validate the current policy before creating a request. Request-options returns an
empty `platforms` list when closed; MCP additionally returns `enabled: false`.
Public environment IDs remain `web` and `web-extension`, not enum names.
Catalog selectors use captured availability per product and hide unavailable pairs.


### Grouped request targets

New clients submit `CreateRequestRequest.targets` as explicit platform/environment
pairs. For example:

```json
{
  "requestType": "PRODUCT_REQUEST_TYPE_ADD",
  "url": "https://metamask.io/",
  "webExtensionUrl": "https://chromewebstore.google.com/detail/metamask/nkbihfbeogaeaoehlefnkodbefgpgknn",
  "targets": [
    { "view": "desktop", "implementation": "web-extension" },
    { "view": "mobile", "implementation": "web" }
  ]
}
```

The server validates each pair against current request availability and persists
these pairs in the typed request details. Approval creates only the selected
pairs: this example does not create a Desktop/Web job. `views` and
`implementations` remain accepted for older clients and represent their Cartesian
product; do not combine them with `targets`. The website source is `url`; extension
jobs use `webExtensionUrl`. For update requests, send the existing `catalogId` and
targets without source links. Select the product first: the platform offers only
its captured platform/environment pairs that are currently enabled for requests.
The server independently rejects targets absent from that product and resolves each environment's source from
that product's published captures and ignores client replacement links. Public
request summaries omit the extension source along with private request details.