commit b6afe941d4eed8f2974e729a362078bd47a8a673 parent 4ba1739f51a97b26583712cd414c17129b9e9783 Author: James Eversole <james@eversole.co> Date: Tue, 13 Oct 2026 17:49:23 -0500 Index compiler toolchains and practical libraries with 16 MiB package inputs Diffstat:
42 files changed, 821 insertions(+), 408 deletions(-)
diff --git a/ext/demo-site/README.md b/ext/demo-site/README.md @@ -62,10 +62,40 @@ adding library pins/assets to this site's existing release/seed pipeline. No library compilation or source dependency resolution happens during site build, seeding, or browser package selection. -### Showcase libraries and parser demo +### Compiler packages, practical libraries and scanning demos The Published packages group also contains these `0.1.0` interfaces: +- **tricu-compiler**: the exact compiler-library bytes already pinned by + `compiler.json`, not a rebuilt or substituted compiler. The supported browser + entrypoint is `source_unit.compileStableModuleDefinitions`; the full retained + library also exposes native tooling helpers. Indexing it does not change + Playground's compiler selection or start compilation. +- **tricu-source-builder**: the exact authenticated shipped + `bootstrap/build-stable.arboricx`, exporting `sourceBuildGraph`. This stable + application compiler resolves module graphs, imports and exports and produces + publication data through the native host. Its store/filesystem actions require + the Haskell host; it is not a standalone browser IO program. Import the downloaded + artifact indexed into a tool store and select its manifest with `--toolchain` + and `--tool-store`, as in [Build and run](../../docs/build-and-run.md.html#explicit-compiler-selection). +- **tricu-math**: binary natural-number arithmetic and ASCII classifiers. Inputs + must be canonical naturals; there is no arbitrary-tree validation on this raw + path. `divMod a 0` returns `(0, a)`. Use the checked base facade for untrusted + numeric inputs. “Native math” names its source module, not a host intrinsic: + these closed Tree Calculus functions are browser-runnable. +- **tricu-lexer**: positioned character scans and token helpers. Callers validate + characters and position naturals. LF advances line; columns/offsets count + source characters, not UTF-8 bytes, tab display width or grapheme clusters. +- **tricu-binary**: checked byte readers, prefix parsing and UTF-8 validation. + Prefix readers validate consumed cells, not an unconsumed suffix; this is not + permission to treat an entire stream as validated after reading one byte. +- **tricu-list-traversal**: early-stopping folds returning state and remaining + input. Stop retains the current cell; malformed tails/decisions halt without + consuming it. Callbacks can still diverge; a remaining suffix is not success. +- **tricu-patterns**: first-match predicate/action rows and thunked conditionals. + These are library callbacks, not language equation patterns or guard syntax. + Unselected actions are not evaluated; a reached malformed row returns Leaf. + - **arboricx-server**: the registry's routes, validation and server function, packaged as closed trees. Serving needs native sockets/filesystem permissions; this is a library interface, not a standalone browser IO program. @@ -89,6 +119,14 @@ remaining input, consumption and commitment, or the accepted names/count. This demo has authenticated reference source, but no browser compilation profile: its native module imports make it **read-only**, not an editable draft. +**Located lexer scans** under Examples is the separate `lexer-demo` package. +`scanWord` scans a lowercase ASCII prefix; `scanWhitespace` scans spaces, LF and +TAB. Records show matched text, remaining input, end line/column, character offset +and whether anything was consumed. Try `"hello,world"`, `"abcé"` and whitespace +containing a newline/tab. Empty prefixes succeed without consuming anything; +these are prefix scans, not whole-input parsers. Its pinned native imports also +make the reference source read-only in the browser. + To reproduce and stage this cohort locally, first build the Haskell host through Nix, then run the opt-in bounded preparation: @@ -99,14 +137,23 @@ python3 ext/demo-site/prepare-showcase.py result/bin/tricu \ ``` Preparation authenticates retained fixtures and requires all recorded working -sources to match them. Five library interfaces retain exact constructor/child-ID +sources to match them. Nine library interfaces retain exact constructor/child-ID roots from those accepted artifacts, renamed without evaluation or expansion. -The formatter and parser demo compile with the authenticated shipped stable -compiler and pinned native dependency interfaces. Real native formatter cases -and independently specified complete parser records must pass before staging; -publication and a clean indexed import must agree on every manifest. Each bundle -also must fit the browser's 1 MiB/100,000-node input bounds. Native commands use -2 GiB RTS bounds, per-command deadlines and a ten-minute whole-job deadline. +The two compiler packages retain their exact accepted bundle bytes. The list +traversal library, formatter and parser/lexer demos compile with the authenticated +shipped stable builder and pinned native dependency interfaces. Acceptance includes +native builds through the published builder (a local dependency succeeds; a +missing dependency fails without emitting a bundle), arithmetic and malformed-byte +controls, predicate dispatch/early-stop folds, real formatter cases, and independent +complete parser/lexer records. Publication and a clean indexed import must agree +on every manifest before staging. + +Ordinary packages must fit the browser's 16 MiB/100,000-node input bounds. +The indexed compiler loads under the same package profile, with no special +name-based exception. Draft compilation separately retains a 2 MiB compiler +input limit. SHA-256, streaming limits, node/export validation and loading +deadlines remain enforced. Native commands use 2 GiB RTS bounds, per-command +deadlines and a ten-minute whole-job deadline. The output directory retains commands, source/artifact identities and acceptance results. `--reuse-directory` permits reuse of its local CAS, while rewriting @@ -116,7 +163,7 @@ publication. Existing release pins cannot be replaced, all assets precede index discovery, and conflicts fail before staging. Native-prepared catalogue entries have `preparation: "native"`; `prepare.py` preserves them and `prepare-index.py` does not assign them a browser compilation environment. -Existing immutable versions and source-only recovery controls are unchanged. +Existing immutable versions are unchanged. ### Published packages and local drafts @@ -182,26 +229,14 @@ compiler; the notice does not prescribe an expected duration. bytes, not a re-encoding, applied result or local draft. Textual exports likewise contain a selected stored definition. These actions and published sharing are disabled while authoring a draft; restore the published source first. Draft bundle -publication/export is not part of this profile. **Download saved source** is a -separate recovery action: it downloads your authored text, not executable trees. - -Draft edits are saved to one `localStorage` recovery slot on this origin, and -saved again synchronously before constructing the compiler Worker. Source, -selected export, arguments, output mode and curated demo context are retained; -compiled heaps and resource settings are not. Recovery stores the last valid -typed arguments; incomplete/invalid literal field text remains only in the current page. The latest draft replaces -that slot. **Recover saved draft** restores an editor with the page default (application -guards off), without running or compiling anything. Curated recovery requires the same published demo hash -and environment identifier still in the catalogue; missing context is an explicit -error, not a fallback to another environment. Saved source can still be downloaded. -**Forget saved draft** removes the recovery copy without clearing the editor. -Later editing or compilation saves it again. - -If storage is blocked, full or invalid, the UI reports that recovery is unavailable. -A failed recovery save is reported in the automatically opened Draft recovery -section, but does not block a requested compilation or open another popup. Local storage is best-effort, not a durable backup: private -browsing, browser cleanup, crashes or other tabs writing the same slot can lose -or replace it. Keep important programs elsewhere too. Nothing is uploaded. +publication/export is not part of this profile. + +Drafts exist only in the open page. There is no autosaving, browser-storage +access, saved-source download or recovery UI. Compilation failure, Stop and +resource-setting changes retain source in the current editor; a page reload +starts fresh from its published selection (or selection link), without restoring, +compiling or running a draft. Copy important source elsewhere before leaving. +Copies stored by older site releases are ignored, not read or restored. ### Arguments, results, and IO @@ -382,14 +417,16 @@ initially and select metadata after entering either package panel. Export has it own definition/format controls, bounded preview, Stop, Copy, and Download actions. Switching panels stops in-flight package work, retaining selections and completed exports; errors appear in the export panel rather than a hidden execution panel. -Compiler downloading/execution starts with Compile. +Compiler downloading/execution for drafts starts with Compile. Explicitly selecting +the indexed compiler package also downloads/verifies its library, without compiling +or executing a program. Both runners start with **Run without optional limits** checked: application node, evaluator-frame, work and active-time execution ceilings are off. Under **Advanced → Optional limits**, uncheck it to enable the editable budgets. Memory budgets count live tree nodes, not megabytes or total browser memory. Garbage collection and Stop remain available with or without limits. Choices are -not persisted across page reloads or recovered from draft storage; reload/recovery +not persisted across page reloads; reload use the limits-off default. Download, format, literal, IO and export checks are independent and remain enabled. @@ -426,7 +463,7 @@ Download and format checks remain enabled when optional execution limits are off Every bundle must match its whole-file SHA-256. A bundle selected for both compiler and ordinary-package roles must satisfy the smaller download cap. -- Ordinary package downloads: 1 MiB (compiler-only profile: 2 MiB); bundles: 100,000 physical indexed nodes, 4,096 exports/roots. +- Ordinary package downloads: 16 MiB (draft-compilation compiler profile: 2 MiB); bundles: 100,000 physical indexed nodes, 4,096 exports/roots. - Graph: 250,000 live nodes (including thunks), 20,000,000 work units, 50,000 evaluation frames when optional limits are enabled, editable under Advanced. These application execution guards are off by default. Loading/observers/codecs consume the session budget. diff --git a/ext/demo-site/bundles/200d0b3d4c58f604a951c986529636f3214c98f36d44b3e867560d74aa4f7a1b.arboricx b/ext/demo-site/bundles/200d0b3d4c58f604a951c986529636f3214c98f36d44b3e867560d74aa4f7a1b.arboricx Binary files differ. diff --git a/ext/demo-site/bundles/260665917d6d973a0a543ea3ba983c47f13e18e60a30d94e502ee8c09ae33653.arboricx b/ext/demo-site/bundles/260665917d6d973a0a543ea3ba983c47f13e18e60a30d94e502ee8c09ae33653.arboricx Binary files differ. diff --git a/ext/demo-site/bundles/50c088ee7b87f229ef37c375e86d32d594610b5ef7cc61ea129351da0ce4b208.arboricx b/ext/demo-site/bundles/50c088ee7b87f229ef37c375e86d32d594610b5ef7cc61ea129351da0ce4b208.arboricx Binary files differ. diff --git a/ext/demo-site/bundles/55565ff3214c7e301d8f8d269121b71173c0e57d89d9436347e5e6dd6bbbef51.arboricx b/ext/demo-site/bundles/55565ff3214c7e301d8f8d269121b71173c0e57d89d9436347e5e6dd6bbbef51.arboricx Binary files differ. diff --git a/ext/demo-site/bundles/5d36d069562ca9b208cfd0aa8df9f0d3b5c39f8b13e070fb892e21b38e32ffb4.arboricx b/ext/demo-site/bundles/5d36d069562ca9b208cfd0aa8df9f0d3b5c39f8b13e070fb892e21b38e32ffb4.arboricx Binary files differ. diff --git a/ext/demo-site/bundles/85184cd7e36328cf195ceb5599aa10c9a9f26a7134f260906379fc8df163531f.arboricx b/ext/demo-site/bundles/85184cd7e36328cf195ceb5599aa10c9a9f26a7134f260906379fc8df163531f.arboricx Binary files differ. diff --git a/ext/demo-site/bundles/d2a20924fd6129069f3b8280289555dcf7d7cc67fb0a5cf488caea0ddaa153cf.arboricx b/ext/demo-site/bundles/d2a20924fd6129069f3b8280289555dcf7d7cc67fb0a5cf488caea0ddaa153cf.arboricx Binary files differ. diff --git a/ext/demo-site/catalogue.json b/ext/demo-site/catalogue.json @@ -453,5 +453,44 @@ } }, "preparation": "native" + }, + { + "id": "lexer", + "preparation": "native", + "title": "Located lexer scans", + "hash": "50c088ee7b87f229ef37c375e86d32d594610b5ef7cc61ea129351da0ce4b208", + "source": "!import prelude *\n!import parser *\n!import lexer *\n!import native_math as N\n!export scanWord\n!export scanWhitespace\n\n-- Positioned prefix scans, not whole-input parsers. The remainder begins\n-- at the first nonmatching character. Positions count source characters,\n-- not UTF-8 bytes or display columns: only LF advances the line.\n-- Native imports are pinned during preparation; browser source is read-only.\nreport reply = (let\n payload = parserReplyPayload reply\n end = snd payload\n in [\n [\"matched\" 0 (fst payload)]\n [\"remaining\" 0 (parserReplyRest reply)]\n [\"line\" 1 (lexerPositionLine end)]\n [\"column\" 1 (lexerPositionColumn end)]\n [\"offset\" 1 (lexerPositionOffset end)]\n [\"consumed\" 2 (parserReplyConsumed reply)]])\n\nscan predicate text = report (lexerMapScan reverse\n (lexerScanWhile text lexerStart [] predicate (state character : pair character state)))\nscanWord text = scan (character : N.inRange character 97 122) text\nscanWhitespace text = scan (character : or? (equal? character 32)\n (or? (equal? character 9) (equal? character 10))) text\n", + "sourceSha256": "5478169b100e087b8f8862418a2cd5d0e593edcc299a122b762842483f4525e0", + "export": "scanWord", + "arguments": [ + { + "type": "string", + "value": "hello,world" + } + ], + "output": "record", + "mode": "pure", + "presets": { + "scanWord": { + "arguments": [ + { + "type": "string", + "value": "hello,world" + } + ], + "output": "record", + "mode": "pure" + }, + "scanWhitespace": { + "arguments": [ + { + "type": "string", + "value": " \n\tword" + } + ], + "output": "record", + "mode": "pure" + } + } } ] diff --git a/ext/demo-site/demos/lexer.tri b/ext/demo-site/demos/lexer.tri @@ -0,0 +1,27 @@ +!import prelude * +!import parser * +!import lexer * +!import native_math as N +!export scanWord +!export scanWhitespace + +-- Positioned prefix scans, not whole-input parsers. The remainder begins +-- at the first nonmatching character. Positions count source characters, +-- not UTF-8 bytes or display columns: only LF advances the line. +-- Native imports are pinned during preparation; browser source is read-only. +report reply = (let + payload = parserReplyPayload reply + end = snd payload + in [ + ["matched" 0 (fst payload)] + ["remaining" 0 (parserReplyRest reply)] + ["line" 1 (lexerPositionLine end)] + ["column" 1 (lexerPositionColumn end)] + ["offset" 1 (lexerPositionOffset end)] + ["consumed" 2 (parserReplyConsumed reply)]]) + +scan predicate text = report (lexerMapScan reverse + (lexerScanWhile text lexerStart [] predicate (state character : pair character state))) +scanWord text = scan (character : N.inRange character 97 122) text +scanWhitespace text = scan (character : or? (equal? character 32) + (or? (equal? character 9) (equal? character 10))) text diff --git a/ext/demo-site/dist/app.js b/ext/demo-site/dist/app.js @@ -2,7 +2,6 @@ import { demoInputLabels } from './demo-inputs.js'; import { argumentEditor } from './arguments-ui.js'; import { createTreeExport } from './export-app.js'; import { activePane } from './tabs.js'; -import { DRAFT_KEY, readDraft, writeDraft } from './draft-storage.js'; import { readSettings } from './limits-ui.js'; import { ActiveClock } from './clock.js'; import { PACKAGE_ACTIVE_MS, EXPORT_ACTIVE_MS, activeMilliseconds, operationFailure, operationTimeout } from './execution.js'; @@ -69,7 +68,6 @@ function controls(working) { hashInput.disabled = busy; el('load-release').disabled = busy || !programPackage(programPicker.value); el('load').disabled = busy; - recoveryButtons(); runButton.disabled = busy || (draft ? !compiled : !loaded); runButton.textContent = draft ? 'Run draft' : 'Run published'; el('draft-run-note').hidden = !draft || compiled; @@ -607,112 +605,6 @@ downloadButton.onclick = () => { newWorker(false, true); } }; -function recoveryButtons() { - try { - const saved = readDraft(localStorage); - el('recover-draft').hidden = !saved; - el('forget-draft').hidden = !saved; - el('download-draft-source').hidden = !saved; - el('recover-draft').disabled = busy || (!!saved?.demo && entries.length === 0); - if (saved && !el('draft-save-status').textContent) - el('draft-save-status').textContent = 'A saved draft is available on this device. Recover it to continue editing; recovery does not run code.'; - } - catch { - el('recover-draft').hidden = true; - el('download-draft-source').hidden = true; - el('forget-draft').hidden = false; - el('draft-save-status').textContent = 'Draft recovery is unavailable. Keep a separate copy of your source. You can try Forget saved draft to remove invalid saved data.'; - } -} -function saveDraft() { - if (!draft) - return true; - try { - writeDraft(localStorage, { schema: 1, source: sourceEditor.value, name: exportsInput.value || preferredName, - arguments: argsInput.value, output: resultCodec.value, mode: mode.value === 'io' ? 'io' : 'pure', - ...(!scratch && selected?.environment !== undefined ? { demo: { hash: selected.hash, environment: selected.environment } } : {}) }); - el('draft-save-status').textContent = 'Draft saved on this device for recovery (latest draft only). Browser storage may be cleared; keep important work elsewhere too.'; - recoveryButtons(); - return true; - } - catch { - el('draft-save-status').closest('details').open = true; - el('draft-save-status').textContent = 'Draft recovery copy could not be saved. You can still compile; keep a separate copy of your source.'; - return false; - } -} -el('recover-draft').onclick = () => { - try { - const saved = readDraft(localStorage); - if (!saved) - return; - const entry = saved.demo ? entries.find(e => e.hash === saved.demo.hash && e.environment === saved.demo.environment) : undefined; - if (saved.demo && !entry) - throw new Error('The saved demo environment is no longer in this catalogue. Use Download saved source to recover your text; it has not been compiled against a substitute environment.'); - const output = outputType(saved.output); - dispose(); - clearOutput(); - clearTreeExport(); - chooseProgram(entry ? `demo:${entry.id}` : '', entry ? releases.find(r => r.demo === entry.id && r.hash === entry.hash)?.version : undefined); - editingSource = true; - el('source-section').open = true; - draft = true; - scratch = !saved.demo; - selected = entry; - originalSource = undefined; - selection = undefined; - loaded = false; - showRelease(entry ? releases.find(r => r.hash === entry.hash) : undefined); - hashInput.value = entry?.hash ?? ''; - el('package-info').textContent = entry ? `Recovered draft based on ${entry.title}. No published executable loaded.` : 'Recovered scratch program. No published package selected.'; - sourceEditor.value = saved.source; - sourceEditor.hidden = false; - el('source-notes').hidden = true; - preferredName = saved.name; - exportsInput.replaceChildren(); - argsInput.value = saved.arguments; - argumentsUI.refresh(true); - resultCodec.value = output; - mode.value = saved.mode; - // Recovery uses the page default, never stored resource settings. - const profile = el('compiler-uncapped'); - profile.checked = true; - profile.dispatchEvent(new Event('change')); - el('source-label').textContent = entry ? `Recovered draft · ${entry.title}` : 'Recovered scratch program · contract helpers'; - controls(false); - el('status').textContent = 'Recovered draft: not compiled'; - } - catch (error) { - el('draft-save-status').textContent = error instanceof Error ? error.message : String(error); - } -}; -el('download-draft-source').onclick = () => { - try { - const saved = readDraft(localStorage); - if (!saved) - return; - const url = URL.createObjectURL(new Blob([saved.source], { type: 'text/plain;charset=utf-8' })); - const anchor = document.createElement('a'); - anchor.href = url; - anchor.download = 'local-draft.tri'; - anchor.click(); - setTimeout(() => URL.revokeObjectURL(url), 1000); - } - catch { - el('draft-save-status').textContent = 'Saved source is unavailable.'; - } -}; -el('forget-draft').onclick = () => { - try { - localStorage.removeItem(DRAFT_KEY); - el('draft-save-status').textContent = 'Saved recovery copy removed. Editor contents are unchanged.'; - recoveryButtons(); - } - catch { - el('draft-save-status').textContent = 'Unable to remove saved draft from local storage.'; - } -}; -recoveryButtons(); el('edit-source').onclick = () => { editingSource = true; el('source-section').open = true; @@ -730,8 +622,6 @@ sourceEditor.oninput = () => { exportsInput.replaceChildren(); draft = scratch || sourceEditor.value !== originalSource; controls(false); - if (draft) - saveDraft(); if (!draft && selection) { preferredExport = preferredName; newWorker(false, true); @@ -759,7 +649,6 @@ function compileDraft() { exportsInput.replaceChildren(); try { sourceText(sourceEditor.value); - saveDraft(); settings = readSettings('compiler'); const current = new Worker(new URL('./compiler-worker.js', import.meta.url), { type: 'module' }); worker = current; @@ -900,7 +789,6 @@ for (const button of document.querySelectorAll('[data-compiler-example]')) argumentsUI.refresh(true); resultCodec.value = example.output; mode.value = example.mode; - saveDraft(); controls(false); el('source-label').textContent = 'Scratch program · contract helpers'; el('status').textContent = 'Ready to compile'; @@ -981,7 +869,6 @@ async function initializePackages() { const indexBytes = await fetchBytes(new URL('../packages.json', import.meta.url), 1_048_576); releases = packageIndex(JSON.parse(new TextDecoder('utf-8', { fatal: true }).decode(indexBytes))); populatePrograms(); - recoveryButtons(); if (!packagePane() || selection || draft) { operation = undefined; return; diff --git a/ext/demo-site/dist/bundle.js b/ext/demo-site/dist/bundle.js @@ -1,6 +1,6 @@ -export const BUNDLE_LIMIT = 1_048_576; +export const BUNDLE_LIMIT = 16_777_216; // Separate bounded profile for the explicitly selected compiler, not packages. -export const COMPILER_BUNDLE_LIMIT = 2 * BUNDLE_LIMIT; +export const COMPILER_BUNDLE_LIMIT = 2_097_152; const utf8 = new TextDecoder('utf-8', { fatal: true }); class Reader { bytes; diff --git a/ext/demo-site/dist/compiler-worker.js b/ext/demo-site/dist/compiler-worker.js @@ -2,10 +2,10 @@ import { environmentProfile } from './packages.js'; import { executionSettings, executionBudget, collectionPolicy } from './execution.js'; import { contractEnvironment } from './contracts.js'; import { Graph } from './graph.js'; -import { COMPILER_BUNDLE_LIMIT, bundleHash, fetchBundle, fetchBytes, loadBundle } from './bundle.js'; +import { COMPILER_BUNDLE_LIMIT, fetchBundle, fetchBytes, loadBundle } from './bundle.js'; import { literals, makeLiteral, outputType, view, list, text } from './codecs.js'; import { runIO } from './io.js'; -import { COMPILER_EXPORT, sourceLiteral, compiledDefinitions } from './compiler.js'; +import { COMPILER_EXPORT, compilerDescriptor, sourceLiteral, compiledDefinitions } from './compiler.js'; const scope = globalThis; const base = new URL('../', import.meta.url); const send = (message) => scope.postMessage(message); @@ -29,11 +29,7 @@ async function compile(source, settings, profileName) { const input = sourceLiteral(graph, source); // Validate before downloading a compiler. const bytes = await fetchBytes(new URL('compiler.json', base), 4096); const raw = JSON.parse(new TextDecoder('utf-8', { fatal: true }).decode(bytes)); - if (!raw || typeof raw !== 'object') - throw new Error('Invalid compiler descriptor'); - const descriptor = raw, hash = bundleHash(descriptor['hash']); - if (descriptor['schema'] !== 1 || descriptor['backend'] !== 'stable' || descriptor['export'] !== COMPILER_EXPORT) - throw new Error('Invalid stable compiler descriptor'); + const { hash } = compilerDescriptor(raw); const bundle = await fetchBundle(new URL(`/_arboricx/bundle/hash/${hash}`, base), hash, COMPILER_BUNDLE_LIMIT); const library = loadBundle(graph, bundle), compiler = library.get(COMPILER_EXPORT); if (compiler === undefined) diff --git a/ext/demo-site/dist/compiler.js b/ext/demo-site/dist/compiler.js @@ -1,5 +1,14 @@ +import { bundleHash } from './bundle.js'; import { text, readText, readList, readNatural } from './codecs.js'; export const COMPILER_EXPORT = 'source_unit.compileStableModuleDefinitions'; +export function compilerDescriptor(raw) { + if (!raw || typeof raw !== 'object' || Array.isArray(raw)) + throw new Error('Invalid compiler descriptor'); + const descriptor = raw; + if (descriptor['schema'] !== 1 || descriptor['backend'] !== 'stable' || descriptor['export'] !== COMPILER_EXPORT) + throw new Error('Invalid stable compiler descriptor'); + return { hash: bundleHash(descriptor['hash']) }; +} export const COMPILER_BUDGET = Object.freeze({ nodes: 6_000_000, fuel: Infinity, stack: 100_000 }); export const COMPILER_COLLECTION = Object.freeze({ threshold: 250_000 }); export const SOURCE_LIMIT = 16_384; diff --git a/ext/demo-site/dist/demo-inputs.js b/ext/demo-site/dist/demo-inputs.js @@ -8,6 +8,7 @@ export function demoInputLabels(id, definition) { case 'shipping': return ['Destination', 'Cart subtotal (cents)', 'Package weight (grams)']; case 'events': return definition === 'vendorB' ? ['Event [customer refunded totalCents]'] : ['Event [eventName customer totalCents]']; case 'registration': return ['Name', 'Email', 'Age', 'Terms accepted (t = false, (t t) = true)']; + case 'lexer': return definition === 'scanWhitespace' ? ['Text with leading whitespace'] : ['Text with a lowercase prefix']; case 'parser': return ['Comma-separated lowercase names']; case 'migration': return ['Profile [name email]']; default: return []; diff --git a/ext/demo-site/dist/draft-storage.js b/ext/demo-site/dist/draft-storage.js @@ -1,38 +0,0 @@ -// One local recovery slot, never executable state or an automatic run request. -// Browser limits are intentionally NOT persisted: recovery starts in Standard. -export const DRAFT_KEY = 'arboricx.local-draft.v1'; -export function savedDraft(raw) { - if (!raw || typeof raw !== 'object' || Array.isArray(raw)) - throw new Error('Invalid saved draft'); - const v = raw; - for (const [key, max] of [['source', 32768], ['name', 256], ['arguments', 16000], ['output', 32]]) - if (typeof v[key] !== 'string' || v[key].length > max) - throw new Error('Invalid saved draft ' + key); - if (v['schema'] !== 1 || (v['mode'] !== 'pure' && v['mode'] !== 'io')) - throw new Error('Invalid saved draft version/mode'); - let demo; - if (v['demo'] !== undefined) { - const d = v['demo']; - if (!d || typeof d !== 'object' || typeof d['hash'] !== 'string' || !/^[0-9a-f]{64}$/u.test(d['hash']) || typeof d['environment'] !== 'string' || !/^[A-Za-z0-9_-]{1,128}$/u.test(d['environment'])) - throw new Error('Invalid saved demo context'); - demo = { hash: d['hash'], environment: d['environment'] }; - } - return { schema: 1, source: v['source'], name: v['name'], arguments: v['arguments'], - output: v['output'], mode: v['mode'], ...(demo ? { demo } : {}) }; -} -export function readDraft(storage) { - const text = storage.getItem(DRAFT_KEY); - if (text === null) - return undefined; - if (text.length > 100000) - throw new Error('Saved draft too large'); - return savedDraft(JSON.parse(text)); -} -export function writeDraft(storage, draft) { - const text = JSON.stringify(savedDraft(draft)); - if (text.length > 100000) - throw new Error('Saved draft too large'); - storage.setItem(DRAFT_KEY, text); - if (storage.getItem(DRAFT_KEY) !== text) - throw new Error('Draft storage verification failed'); -} diff --git a/ext/demo-site/index.html b/ext/demo-site/index.html @@ -43,7 +43,6 @@ <section class="source-panel compiler-source" aria-labelledby="source-title"><div class="panel-heading"><h3 id="source-title">Package information</h3><span id="source-label">tricu programs</span><button id="edit-source" type="button" disabled>Edit source</button></div> <p id="source-availability" class="hint" hidden></p> <p id="compiler-disclosure" class="hint" hidden>Compilation uses a JavaScript Tree Calculus reducer and can take time or extra memory. Use the <a href="https://git.eversole.co/James/tricu/file/docs/build-and-run.md.html">native Haskell tools</a> for larger builds. Set optional limits under Advanced; Stop cancels compilation and keeps your source.</p> - <details class="draft-recovery"><summary>Draft recovery</summary><p id="draft-save-status" class="hint" role="status"></p><button type="button" id="recover-draft" hidden>Recover saved draft</button><button type="button" id="download-draft-source" hidden>Download saved source</button><button type="button" id="forget-draft" hidden>Forget saved draft</button></details> <p id="authorship" role="status">Published package</p> <label class="sr-only" for="source-editor">Editable demo source (up to 16,384 characters)</label> <textarea id="source-editor" rows="16" maxlength="32768" spellcheck="false" readonly hidden></textarea> @@ -79,7 +78,7 @@ <p>Optional limits: <strong id="package-resource-mode">Off</strong></p> <label><input id="uncapped" type="checkbox" checked aria-describedby="package-resource-warning"> Run without optional limits</label> <p id="package-resource-warning" class="hint">Uncheck to set limits below. Limits can stop a job before it uses more resources; Stop remains available either way.</p> - </div><h3>Session settings</h3><div id="package-limits-defaults"></div><p class="hint">Memory is measured in live tree nodes, not megabytes. Garbage collection stays active with or without these limits. Bundle validation and IO format checks always apply.</p><p class="hint">Package input: 1 MiB bundle and 100,000 stored nodes; loading allows 60 seconds, including transfer and processing. IO: up to 128 actions, 256 characters per line and 16,384 output characters. Tree size counts every occurrence, including shared branches.</p></div></details> + </div><h3>Session settings</h3><div id="package-limits-defaults"></div><p class="hint">Memory is measured in live tree nodes, not megabytes. Garbage collection stays active with or without these limits. Bundle validation and IO format checks always apply.</p><p class="hint">Package input: 16 MiB bundle and 100,000 stored nodes; loading allows 60 seconds, including transfer and processing. IO: up to 128 actions, 256 characters per line and 16,384 output characters. Tree size counts every occurrence, including shared branches.</p></div></details> <details id="compiler-settings" class="execution-limits" hidden><summary id="compiler-limits-summary">Optional resource limits</summary><div class="limits-content"><div id="compiler-resource-choice" class="resource-choice" hidden> <p>Optional limits: <strong id="compiler-resource-mode">Off</strong></p> <label><input id="compiler-uncapped" type="checkbox" checked aria-describedby="compiler-resource-warning"> Run without optional limits</label> diff --git a/ext/demo-site/lexer-demo.json b/ext/demo-site/lexer-demo.json @@ -0,0 +1,145 @@ +{ + "id": "lexer", + "hash": "50c088ee7b87f229ef37c375e86d32d594610b5ef7cc61ea129351da0ce4b208", + "mode": "pure", + "compilerSha256": "5d36d069562ca9b208cfd0aa8df9f0d3b5c39f8b13e070fb892e21b38e32ffb4", + "sourceSha256": "5478169b100e087b8f8862418a2cd5d0e593edcc299a122b762842483f4525e0", + "haskellCases": [ + { + "export": "scanWord", + "arguments": [ + "220202021020210210022102021020202102100220202102102021021002202021021020210210022102102102102021021002202021021020210022102102102021021021002210210210210202102100220210202021021021002202021021020210210022020210202021021000" + ], + "input": null, + "terminal": "", + "result": "222210202102102021021002210202020202102100220202102021021021002210210202020210210022020202102021021002210202102020210210022020210202021021000202220202021020210210022102021020202102100220202102102021021002202021021020210210022102102102102021021000022220210202021021021002210202102020210210022102021021020210210022102020202021021002210202021020210210022021021021020210210022102020210202102100220210210210202102100221021021020202102100020222020210210202100221021021020210210210022102102102102021021002202102020210210210022020210210202102100220202102020210210000222202021021020210210022102020210202102100220210210210202102100221020210202021021000221002210002222102102020202102100221021021021020210210022020210210202102100221020210202102102100221020210210202102100220210210210202102100022100220210210002222102102102102021021002202102102020210210022021021020202102100221021020202102102100221020210202021021002202021020210210210002210022102021000222210210202020210210022102102102102021021002202102102102021021002210210202021021021002210202102021021021002210202102102021021002210202102020210210022020210202021021000220210021000", + "record": { + "matched": "hello", + "remaining": ",world", + "line": 1, + "column": 6, + "offset": 5, + "consumed": true + } + }, + { + "export": "scanWord", + "arguments": [ + "0" + ], + "input": null, + "terminal": "", + "result": "22221020210210202102100221020202020210210022020210202102102100221021020202021021002202020210202102100221020210202021021002202021020202102100020200222202102020210210210022102021020202102100221020210210202102100221020202020210210022102020210202102100220210210210202102100221020202102021021002202102102102021021002210210210202021021000202002222020210210202102100221020202102021021002202102102102021021002210202102020210210002210022100022221021020202021021002210210210210202102100220202102102021021002210202102021021021002210202102102021021002202102102102021021000221002210002222102102102102021021002202102102020210210022021021020202102100221021020202102102100221020210202021021002202021020210210210002210020022221021020202021021002210210210210202102100220210210210202102100221021020202102102100221020210202102102100221020210210202102100221020210202021021002202021020202102100022021002000", + "record": { + "matched": "", + "remaining": "", + "line": 1, + "column": 1, + "offset": 0, + "consumed": false + } + }, + { + "export": "scanWord", + "arguments": [ + "22020202102020210022102021020202102100220202102102021021002202021021020210210022102102102102021021000" + ], + "input": null, + "terminal": "", + "result": "222210202102102021021002210202020202102100220202102021021021002210210202020210210022020202102021021002210202102020210210022020210202021021000202002222021020202102102100221020210202021021002210202102102021021002210202020202102100221020202102021021002202102102102021021002210202021020210210022021021021020210210022102102102020210210002022202020210202021002210202102020210210022020210210202102100220202102102021021002210210210210202102100002222020210210202102100221020202102021021002202102102102021021002210202102020210210002210022100022221021020202021021002210210210210202102100220202102102021021002210202102021021021002210202102102021021002202102102102021021000221002210002222102102102102021021002202102102020210210022021021020202102100221021020202102102100221020210202021021002202021020210210210002210020022221021020202021021002210210210210202102100220210210210202102100221021020202102102100221020210202102102100221020210210202102100221020210202021021002202021020202102100022021002000", + "record": { + "matched": "", + "remaining": "Hello", + "line": 1, + "column": 1, + "offset": 0, + "consumed": false + } + }, + { + "export": "scanWord", + "arguments": [ + "2210202020202102100220210202020210210022102102020202102100221020202102021021021000" + ], + "input": null, + "terminal": "", + "result": "2222102021021020210210022102020202021021002202021020210210210022102102020202102100220202021020210210022102021020202102100220202102020210210002022210202020202102100220210202020210210022102102020202102100002222021020202102102100221020210202021021002210202102102021021002210202020202102100221020202102021021002202102102102021021002210202021020210210022021021021020210210022102102102020210210002022210202021020210210210000222202021021020210210022102020210202102100220210210210202102100221020210202021021000221002210002222102102020202102100221021021021020210210022020210210202102100221020210202102102100221020210210202102100220210210210202102100022100220202100022221021021021020210210022021021020202102100220210210202021021002210210202021021021002210202102020210210022020210202102102100022100221021000222210210202020210210022102102102102021021002202102102102021021002210210202021021021002210202102021021021002210202102102021021002210202102020210210022020210202021021000220210021000", + "record": { + "matched": "abc", + "remaining": "\u00e9", + "line": 1, + "column": 4, + "offset": 3, + "consumed": true + } + }, + { + "export": "scanWhitespace", + "arguments": [ + "220202020202100220210202100221020202100221021021020210210210022102102102102021021002202102020210210210022020210202021021000" + ], + "input": null, + "terminal": "", + "result": "222210202102102021021002210202020202102100220202102021021021002210210202020210210022020202102021021002210202102020210210022020210202021021000202220202020202100220210202100221020202100002222021020202102102100221020210202021021002210202102102021021002210202020202102100221020202102021021002202102102102021021002210202021020210210022021021021020210210022102102102020210210002022210210210202102102100221021021021020210210022021020202102102100220202102020210210000222202021021020210210022102020210202102100220210210210202102100221020210202021021000221002202100022221021020202021021002210210210210202102100220202102102021021002210202102021021021002210202102102021021002202102102102021021000221002202100022221021021021020210210022021021020202102100220210210202021021002210210202021021021002210202102020210210022020210202102102100022100221021000222210210202020210210022102102102102021021002202102102102021021002210210202021021021002210202102021021021002210202102102021021002210202102020210210022020210202021021000220210021000", + "record": { + "matched": " \n\t", + "remaining": "word", + "line": 2, + "column": 2, + "offset": 3, + "consumed": true + } + }, + { + "export": "scanWhitespace", + "arguments": [ + "2202102021002202102021002210202102020210210022021021021020210210022020210202021021000" + ], + "input": null, + "terminal": "", + "result": "22221020210210202102100221020202020210210022020210202102102100221021020202021021002202020210202102100221020210202021021002202021020202102100020222021020210022021020210000222202102020210210210022102021020202102100221020210210202102100221020202020210210022102020210202102100220210210210202102100221020202102021021002202102102102021021002210210210202021021000202221020210202021021002202102102102021021002202021020202102100002222020210210202102100221020202102021021002202102102102021021002210202102020210210002210022102100022221021020202021021002210210210210202102100220202102102021021002210202102021021021002210202102102021021002202102102102021021000221002210002222102102102102021021002202102102020210210022021021020202102100221021020202102102100221020210202021021002202021020210210210002210022021000222210210202020210210022102102102102021021002202102102102021021002210210202021021021002210202102021021021002210202102102021021002210202102020210210022020210202021021000220210021000", + "record": { + "matched": "\n\n", + "remaining": "end", + "line": 3, + "column": 1, + "offset": 2, + "consumed": true + } + }, + { + "export": "scanWhitespace", + "arguments": [ + "2210202102100220210202100221021021020210210210022102102102102021021002202102020210210210022020210202021021000" + ], + "input": null, + "terminal": "", + "result": "22221020210210202102100221020202020210210022020210202102102100221021020202021021002202020210202102100221020210202021021002202021020202102100020200222202102020210210210022102021020202102100221020210210202102100221020202020210210022102020210202102100220210210210202102100221020202102021021002202102102102021021002210210210202021021000202221020210210022021020210022102102102021021021002210210210210202102100220210202021021021002202021020202102100002222020210210202102100221020202102021021002202102102102021021002210202102020210210002210022100022221021020202021021002210210210210202102100220202102102021021002210202102021021021002210202102102021021002202102102102021021000221002210002222102102102102021021002202102102020210210022021021020202102100221021020202102102100221020210202021021002202021020210210210002210020022221021020202021021002210210210210202102100220210210210202102100221021020202102102100221020210202102102100221020210210202102100221020210202021021002202021020202102100022021002000", + "record": { + "matched": "", + "remaining": "\r\nword", + "line": 1, + "column": 1, + "offset": 0, + "consumed": false + } + }, + { + "export": "scanWhitespace", + "arguments": [ + "221020202102021021021000" + ], + "input": null, + "terminal": "", + "result": "2222102021021020210210022102020202021021002202021020210210210022102102020202102100220202021020210210022102021020202102100220202102020210210002020022220210202021021021002210202102020210210022102021021020210210022102020202021021002210202021020210210022021021021020210210022102020210202102100220210210210202102100221021021020202102100020222102020210202102102100002222020210210202102100221020202102021021002202102102102021021002210202102020210210002210022100022221021020202021021002210210210210202102100220202102102021021002210202102021021021002210202102102021021002202102102102021021000221002210002222102102102102021021002202102102020210210022021021020202102100221021020202102102100221020210202021021002202021020210210210002210020022221021020202021021002210210210210202102100220210210210202102100221021020202102102100221020210202102102100221020210210202102100221020210202021021002202021020202102100022021002000", + "record": { + "matched": "", + "remaining": "\u00e9", + "line": 1, + "column": 1, + "offset": 0, + "consumed": false + } + } + ] +} diff --git a/ext/demo-site/packages.json b/ext/demo-site/packages.json @@ -196,6 +196,63 @@ "manifest": "e6edf23a694a43f5b476c5fbb303ed722e60ead568388766a99867f588b7272a", "title": "Parser combinators: complete input, prefixes and committed errors", "demo": "parser" + }, + { + "name": "tricu-compiler", + "version": "0.1.0", + "hash": "eddb9464fdf42e5126b239a9481ad5915c6152ddf7f396278444894452d16a1a", + "manifest": "feaf7c753b7e4dca1f5512efb701383558e7e327ac824311599767213583c519", + "title": "Compiler library (stable browser source-unit API; native helpers)" + }, + { + "name": "tricu-source-builder", + "version": "0.1.0", + "hash": "5d36d069562ca9b208cfd0aa8df9f0d3b5c39f8b13e070fb892e21b38e32ffb4", + "manifest": "83cfddf90747db82b693734b95469eeec2f4a783576886dcddef41515ec48588", + "title": "Stable native application compiler (modules, imports and publication)" + }, + { + "name": "tricu-math", + "version": "0.1.0", + "hash": "d2a20924fd6129069f3b8280289555dcf7d7cc67fb0a5cf488caea0ddaa153cf", + "manifest": "962c68b1f228cae96d5eb7423b44f77ab2bd8c1bec03b790d6eebe90bc048da6", + "title": "Binary arithmetic and ASCII classifiers (canonical naturals only)" + }, + { + "name": "tricu-lexer", + "version": "0.1.0", + "hash": "85184cd7e36328cf195ceb5599aa10c9a9f26a7134f260906379fc8df163531f", + "manifest": "7d9d5a8b46b7c2cec7a19ca4b430b249ed13f7c3133c2a395a07767ed4cde612", + "title": "Located character scans and token positions" + }, + { + "name": "tricu-binary", + "version": "0.1.0", + "hash": "200d0b3d4c58f604a951c986529636f3214c98f36d44b3e867560d74aa4f7a1b", + "manifest": "89273ee209a5e49b57193fab642f25f51a46885b11ce10af3951d8ef4d321bfe", + "title": "Checked byte readers, prefix parsing and UTF-8 validation" + }, + { + "name": "tricu-patterns", + "version": "0.1.0", + "hash": "55565ff3214c7e301d8f8d269121b71173c0e57d89d9436347e5e6dd6bbbef51", + "manifest": "16d074eda39e963b6aa931d9aa1cc7b01d94bf3203a9a75c26d194f7026f2232", + "title": "Ordered predicate/action dispatch and thunked conditionals" + }, + { + "name": "tricu-list-traversal", + "version": "0.1.0", + "hash": "260665917d6d973a0a543ea3ba983c47f13e18e60a30d94e502ee8c09ae33653", + "manifest": "37aa135a7d6e7365d632e65d5aaf12d18e734cee1ef1f1b47d5a730fddf6592e", + "title": "Early-stopping list folds with retained input suffixes" + }, + { + "name": "lexer-demo", + "version": "0.1.0", + "hash": "50c088ee7b87f229ef37c375e86d32d594610b5ef7cc61ea129351da0ce4b208", + "manifest": "ebc3b77fbbdd82068bb3fb407a597bb3fe54144ca31f7dc32c38720783c7e326", + "title": "Located scans: matched prefix, remainder and character positions", + "demo": "lexer" } ] } diff --git a/ext/demo-site/prepare-index.py b/ext/demo-site/prepare-index.py @@ -67,7 +67,7 @@ def main(): file = tmp / 'environment.arboricx' run('arboricx', 'export', '--store', store, '--module', pin, '--output', file) data = file.read_bytes() - assert len(data) <= 1_048_576 + assert len(data) <= 16_777_216 support_hash = digest(data) profiles = dict(schema=1, compiler=compiler['hash'], profiles={ 'empty': None, diff --git a/ext/demo-site/prepare-showcase.py b/ext/demo-site/prepare-showcase.py @@ -1,5 +1,5 @@ #!/usr/bin/env python3 -"""Opt-in Haskell publication of accepted library roots and a native parser demo. +"""Opt-in Haskell publication of accepted library roots and native parser/lexer demos. No fixture regeneration or public-network access. Requires current sources to match authenticated retained fixtures. Stages only after bounded acceptance. @@ -30,6 +30,10 @@ LIBRARIES = [ ('tricu-intensional-contracts', 'intensional', 'support', 'Intensional contracts: tree shapes and numeric structure'), ('tricu-graphs', 'graph', 'compiler', 'Dependency ordering and cycle witnesses (validated small graphs)'), ('tricu-parser', 'parser', 'compiler', 'Parser combinators: diagnostics, consumption and commitment'), + ('tricu-math', 'native_math', 'compiler', 'Binary arithmetic and ASCII classifiers (canonical naturals only)'), + ('tricu-lexer', 'lexer', 'compiler', 'Located character scans and token positions'), + ('tricu-binary', 'binary', 'support', 'Checked byte readers, prefix parsing and UTF-8 validation'), + ('tricu-patterns', 'patterns', 'compiler', 'Ordered predicate/action dispatch and thunked conditionals'), ] @@ -64,7 +68,24 @@ def record_tree(record): for key, value in record.items()]) +def lexer_cases(): + # Independent character-counting expectations, including LF, tabs and Unicode. + return [ + ('scanWord', 'hello,world', dict(matched='hello', remaining=',world', line=1, column=6, offset=5, consumed=True)), + ('scanWord', '', dict(matched='', remaining='', line=1, column=1, offset=0, consumed=False)), + ('scanWord', 'Hello', dict(matched='', remaining='Hello', line=1, column=1, offset=0, consumed=False)), + ('scanWord', 'abcé', dict(matched='abc', remaining='é', line=1, column=4, offset=3, consumed=True)), + ('scanWhitespace', ' \n\tword', dict(matched=' \n\t', remaining='word', line=2, column=2, offset=3, consumed=True)), + ('scanWhitespace', '\n\nend', dict(matched='\n\n', remaining='end', line=3, column=1, offset=2, consumed=True)), + ('scanWhitespace', '\r\nword', dict(matched='', remaining='\r\nword', line=1, column=1, offset=0, consumed=False)), + ('scanWhitespace', 'é', dict(matched='', remaining='é', line=1, column=1, offset=0, consumed=False)), + ] + + def stage(site, artifacts, releases, demo, goldens): + demos = demo if isinstance(demo, list) else [demo] + vectors = goldens if isinstance(goldens, list) else [goldens] + if [e['id'] for e in demos] != [g['id'] for g in vectors]: raise ValueError('Demo/golden identity mismatch') index = json.loads((site / 'packages.json').read_text()) catalogue = json.loads((site / 'catalogue.json').read_text()) if index.get('schema') != 1: raise ValueError('Unsupported package index') @@ -76,9 +97,10 @@ def stage(site, artifacts, releases, demo, goldens): previous = next((r for r in index['releases'] if (r['name'], r['version']) == (release['name'], release['version'])), None) if previous and previous != release: raise ValueError('Immutable release conflict: ' + release['name']) if not previous: index['releases'].append(release) - previous = next((e for e in catalogue if e['id'] == demo['id']), None) - if previous and previous != demo: raise ValueError('Existing demo differs; release explicitly instead of overwriting') - if not previous: catalogue.append(demo) + for demo in demos: + previous = next((e for e in catalogue if e['id'] == demo['id']), None) + if previous and previous != demo: raise ValueError('Existing demo differs; release explicitly instead of overwriting') + if not previous: catalogue.append(demo) for pin, path in artifacts.items(): data = path.read_bytes() if hashlib.sha256(data).hexdigest() != pin: raise ValueError('Artifact changed before staging') @@ -90,7 +112,7 @@ def stage(site, artifacts, releases, demo, goldens): destination = site / 'bundles' / (pin + '.arboricx') if not destination.exists(): with destination.open('xb') as output: output.write(path.read_bytes()) - for name, value in [('packages.json', index), ('catalogue.json', catalogue), ('parser-demo.json', goldens)]: + for name, value in [('packages.json', index), ('catalogue.json', catalogue)] + [(g['id'] + '-demo.json', g) for g in vectors]: with tempfile.NamedTemporaryFile(dir=site, delete=False) as output: temporary = Path(output.name) output.write((json.dumps(value, indent=2) + '\n').encode()) @@ -108,6 +130,15 @@ def main(): parser.add_argument('--stage-site', action='store_true', help='After acceptance, stage into this static site') args = parser.parse_args() metadata = verify() + descriptor = json.loads((HERE / 'compiler.json').read_text()) + compiler_library = ROOT / 'test/fixtures/compiler-library.arboricx' + if (descriptor['schema'], descriptor['backend'], descriptor['export']) != (1, 'stable', 'source_unit.compileStableModuleDefinitions'): + raise ValueError('Invalid browser compiler descriptor') + if hashlib.sha256(compiler_library.read_bytes()).hexdigest() != descriptor['hash']: + raise ValueError('Browser compiler differs from authenticated compiler fixture') + if compiler_library.stat().st_size > 2_097_152: raise ValueError('Browser compiler exceeds draft-compilation profile') + if (HERE / 'bundles' / (descriptor['hash'] + '.arboricx')).read_bytes() != compiler_library.read_bytes(): + raise ValueError('Browser compiler asset differs from authenticated compiler fixture') mismatches = current_mismatches(ROOT / 'test/fixtures', ROOT) if mismatches: raise ValueError('Current sources differ from retained fixtures: ' + str(mismatches)) root = args.directory.resolve(); root.mkdir(parents=True, exist_ok=args.reuse_directory) @@ -117,7 +148,7 @@ def main(): report = dict(complete=False, compiler=metadata['sha256'], hostSha256=hashlib.sha256(host.read_bytes()).hexdigest(), commands=[], releases=[], sourceSha256={}, fixtureSha256={}) def save(): (root / 'report.json').write_text(json.dumps(report, indent=2) + '\n') - def run(*words, seconds=45, strip=True): + def run(*words, seconds=60, strip=True, success=True): remaining = 600 - (time.monotonic() - started) if remaining <= 0: raise TimeoutError('Showcase preparation whole-job deadline exceeded') command = [str(host), *map(str, words), '+RTS', '-N1', '-M2048m', '-RTS'] @@ -126,7 +157,7 @@ def main(): report['commands'].append(dict(argv=command[1:], seconds=time.monotonic()-before, code=reply.returncode, stdout=reply.stdout.decode(), stderr=reply.stderr.decode())) save() - if reply.returncode: raise RuntimeError(report['commands'][-1]) + if (reply.returncode == 0) != success: raise RuntimeError(report['commands'][-1]) return reply.stdout.decode().strip() if strip else reply.stdout.decode() def imported(bundle, target=store): reply = run('arboricx', 'import', '--indexed', '--store', target, '--file', bundle) @@ -143,7 +174,8 @@ def main(): artifacts = {} def published(name, title, bundle, demo=None): data = bundle.read_bytes() - if not 0 < len(data) <= 1048576 or len(read(bundle)[2]) > 100000: + limit = 16_777_216 + if not 0 < len(data) <= limit or len(read(bundle)[2]) > 100000: raise ValueError('Package exceeds browser input bounds: ' + name) result = run('package', 'publish', '--root', root / 'registry', '--name', name, '--version', '0.1.0', '--file', bundle) @@ -155,13 +187,41 @@ def main(): if demo: release['demo'] = demo artifacts[pin] = bundle; report['releases'].append(release) return release + published('tricu-compiler', 'Compiler library (stable browser source-unit API; native helpers)', compiler_library) + builder = published('tricu-source-builder', 'Stable native application compiler (modules, imports and publication)', ROOT / metadata['bundle']) + report['sourceBuilderSha256'] = metadata['sha256'] + report['browserCompilerSha256'] = descriptor['hash'] for name, alias, cohort, title in LIBRARIES: published(name, title, extracted(alias, cohort)) - for alias in ['prelude', 'parser', 'source_lexer', 'source_equations', 'source_ast', 'native_math']: + def tree_list(values): return ''.join('2' + tree for tree in values) + '0' + def result_tree(success, value, rest): return '2' + ('10' if success else '0') + '2' + value + rest + checks = [ + ('tricu-math', 'add', [of_number(123456789), of_number(11)], of_number(123456800)), + ('tricu-math', 'divMod', [of_number(100), of_number(7)], '2' + of_number(14) + of_number(2)), + ('tricu-math', 'divMod', [of_number(100), of_number(0)], '2' + of_number(0) + of_number(100)), + ('tricu-binary', 'readU8', [tree_list([of_number(65), of_number(66)])], + result_tree(True, of_number(65), tree_list([of_number(66)]))), + ('tricu-binary', 'readU8', [tree_list([of_number(256)])], + result_tree(False, of_number(15), tree_list([of_number(256)]))), + ('tricu-binary', 'binaryUtf8?', [tree_list([of_number(195), of_number(169)])], '10'), + ('tricu-binary', 'binaryUtf8?', [tree_list([of_number(192), of_number(175)])], '0'), + ] + report['libraryCases'] = [] + for name, export, trees, expected in checks: + manifest = next(r['manifest'] for r in report['releases'] if r['name'] == name) + arguments = [] + for i, tree in enumerate(trees): + path = root / ('library-arg-' + str(i) + '.tree'); path.write_bytes(bytes(map(int, tree))) + arguments.extend(['--arg-tree-file', path]) + actual = run('arboricx', 'run', '--store', root / 'consumer', '--module', manifest, + '--name', export, *arguments, '--format', 'ternary') + assert actual == expected, (name, export, actual[:200], expected) + report['libraryCases'].append(dict(name=name, export=export, arguments=trees, result=actual)) + for alias in ['prelude', 'parser', 'lexer', 'source_lexer', 'source_equations', 'source_ast', 'native_math']: pin = imported(extracted(alias, 'compiler')) (workspace / (alias + '.tri')).write_text('!export * from ' + pin + '\n') for alias, source in [('recursion', ROOT / 'lib/recursion/list.tri'), ('formatter', ROOT / 'lib/tools/format-tri.tri'), - ('parser_demo', HERE / 'demos/parser.tri')]: + ('parser_demo', HERE / 'demos/parser.tri'), ('lexer_demo', HERE / 'demos/lexer.tri')]: data = source.read_bytes(); (workspace / (alias + '.tri')).write_bytes(data) report['sourceSha256'][str(source.relative_to(ROOT))] = hashlib.sha256(data).hexdigest() (workspace / 'recursion_support.tri').write_text('!import prelude as B\n!export pair\npair = B.pair\n') @@ -174,6 +234,38 @@ def main(): '--toolchain', compiler, '--tool-store', root / 'tools', '--output', bundle, '--compile-timeout', '180', seconds=195) return bundle, imported(bundle) + # Exercise the actual published builder through its native CLI ABI, with a + # local dependency and a fresh store. No sockets, public registry or fallback. + (workspace / 'probe_dep.tri').write_text('!export answer\nanswer = 42\n') + (workspace / 'builder_probe.tri').write_text('!import probe_dep as D\n!export main\nmain = D.answer\n') + probe = root / 'builder-probe.arboricx' + run('arboricx', 'compile', '--file', workspace / 'builder_probe.tri', '--store', root / 'builder-probe-store', + '--toolchain', builder['manifest'], '--tool-store', root / 'consumer', '--output', probe, + '--compile-timeout', '45', seconds=60) + probe_manifest = imported(probe, root / 'builder-probe-consumer') + assert run('arboricx', 'run', '--store', root / 'builder-probe-consumer', '--module', probe_manifest, + '--name', 'main', '--format', 'number') == '42' + (workspace / 'missing_dep.tri').write_text('!import absent_showcase_dependency as M\n!export main\nmain = M.answer\n') + run('arboricx', 'compile', '--file', workspace / 'missing_dep.tri', '--store', root / 'builder-probe-store', + '--toolchain', builder['manifest'], '--tool-store', root / 'consumer', '--output', root / 'must-not-exist.arboricx', + '--compile-timeout', '45', seconds=60, success=False) + assert not (root / 'must-not-exist.arboricx').exists(), 'failed build emitted executable' + recursion_bundle, _ = compile_unit('recursion') + published('tricu-list-traversal', 'Early-stopping list folds with retained input suffixes', recursion_bundle) + patterns = next(r for r in report['releases'] if r['name'] == 'tricu-patterns') + traversal = next(r for r in report['releases'] if r['name'] == 'tricu-list-traversal') + assert imported(artifacts[patterns['hash']]) == patterns['manifest'] + assert imported(artifacts[traversal['hash']]) == traversal['manifest'] + (workspace / 'published_patterns.tri').write_text('!export * from ' + patterns['manifest'] + '\n') + (workspace / 'published_traversal.tri').write_text('!export * from ' + traversal['manifest'] + '\n') + probe_source = (HERE / 'test/showcase-library-probe.tri').read_bytes() + report['sourceSha256']['ext/demo-site/test/showcase-library-probe.tri'] = hashlib.sha256(probe_source).hexdigest() + (workspace / 'library_probe.tri').write_bytes(probe_source) + library_probe, library_manifest = compile_unit('library_probe') + expected = tree_list([of_number(13), of_number(99), '0', '2' + of_number(3) + tree_list([of_number(0), of_number(9)])]) + actual = run('arboricx', 'run', '--store', store, '--module', library_manifest, '--name', 'main', '--format', 'ternary') + assert actual == expected, 'published dispatch/list traversal probe' + report['libraryProbe'] = dict(result=actual, bundleSha256=hashlib.sha256(library_probe.read_bytes()).hexdigest()) formatter, formatter_manifest = compile_unit('formatter') # Independent real formatter cases, no file effects or diverging probe needed. for label, before, expected in json.loads((ROOT / 'test/formatter_cases.json').read_text()): @@ -197,8 +289,25 @@ def main(): presets={name: preset for name in ['parseNames', 'parsePrefix']}) goldens = dict(id='parser', hash=release['hash'], mode='pure', compilerSha256=metadata['sha256'], sourceSha256=hashlib.sha256(source).hexdigest(), haskellCases=goldens) + demos, vectors = [demo], [goldens] + lexer_bundle, lexer_manifest = compile_unit('lexer_demo') + cases = [] + for export, text, expected in lexer_cases(): + actual = run('arboricx', 'run', '--store', store, '--module', lexer_manifest, + '--name', export, '--arg', text, '--format', 'ternary') + assert actual == record_tree(expected), (export, text, actual[:200], expected) + cases.append(dict(export=export, arguments=[of_string(text)], input=None, terminal='', result=actual, record=expected)) + release = published('lexer-demo', 'Located scans: matched prefix, remainder and character positions', lexer_bundle, 'lexer') + source = (HERE / 'demos/lexer.tri').read_bytes() + word = dict(arguments=[dict(type='string', value='hello,world')], output='record', mode='pure') + whitespace = dict(arguments=[dict(type='string', value=' \n\tword')], output='record', mode='pure') + demos.append(dict(id='lexer', preparation='native', title='Located lexer scans', hash=release['hash'], source=source.decode(), + sourceSha256=hashlib.sha256(source).hexdigest(), export='scanWord', **word, + presets=dict(scanWord=word, scanWhitespace=whitespace))) + vectors.append(dict(id='lexer', hash=release['hash'], mode='pure', compilerSha256=metadata['sha256'], + sourceSha256=hashlib.sha256(source).hexdigest(), haskellCases=cases)) report['complete'] = True; report['elapsedSeconds'] = time.monotonic()-started; save() - if args.stage_site: stage(HERE, artifacts, report['releases'], demo, goldens) + if args.stage_site: stage(HERE, artifacts, report['releases'], demos, vectors) print('Accepted', len(report['releases']), 'releases in', round(report['elapsedSeconds'], 1), 'seconds') diff --git a/ext/demo-site/release.py b/ext/demo-site/release.py @@ -11,14 +11,14 @@ import shutil def dependencies(catalogue, compiler, bundles, index, environments): uses = {} limits = {} - entries = [(entry, 1_048_576) for entry in json.loads(catalogue.read_text())] + entries = [(entry, 16_777_216) for entry in json.loads(catalogue.read_text())] compiler = json.loads(compiler.read_text()) entries.append((dict(compiler, id='compiler'), 2 * 1_048_576)) if environments['schema'] != 1 or environments['compiler'] != compiler['hash']: raise ValueError('Environment/compiler mismatch') for name, profile in environments['profiles'].items(): if profile is not None: - entries.append((dict(hash=profile['hash'], id='environment:' + name), 1_048_576)) + entries.append((dict(hash=profile['hash'], id='environment:' + name), 16_777_216)) if index['schema'] != 1 or len(index['releases']) > 4096: raise ValueError('Invalid package index') seen = set() @@ -29,7 +29,7 @@ def dependencies(catalogue, compiler, bundles, index, environments): if (name, version) in seen or not re.fullmatch('[0-9a-f]{64}', release['manifest']): raise ValueError('Invalid/duplicate release') seen.add((name, version)) - entries.append((dict(hash=release['hash'], id='release:' + name + '@' + version), 1_048_576)) + entries.append((dict(hash=release['hash'], id='release:' + name + '@' + version), 16_777_216)) for entry, limit in entries: digest = entry['hash'] if not isinstance(digest, str) or not re.fullmatch('[0-9a-f]{64}', digest): diff --git a/ext/demo-site/seed.py b/ext/demo-site/seed.py @@ -50,7 +50,7 @@ def seed(pack, root): or any(not isinstance(use, str) or not use for use in uses) or len(set(uses)) != len(uses)): raise ValueError('Invalid dependency uses') - limit = 2 * 1_048_576 if uses == ['compiler'] else 1_048_576 + limit = 2_097_152 if 'compiler' in uses else 16_777_216 if (not isinstance(digest, str) or not re.fullmatch('[0-9a-f]{64}', digest) or digest in seen or type(size) is not int or not 0 < size <= limit): raise ValueError('Invalid dependency entry') diff --git a/ext/demo-site/src/app.ts b/ext/demo-site/src/app.ts @@ -2,7 +2,6 @@ import { demoInputLabels } from './demo-inputs.js'; import { argumentEditor } from './arguments-ui.js'; import { createTreeExport } from './export-app.js'; import { activePane } from './tabs.js'; -import { DRAFT_KEY, readDraft, writeDraft } from './draft-storage.js'; import { readSettings } from './limits-ui.js'; import { ActiveClock } from './clock.js'; import { PACKAGE_ACTIVE_MS, EXPORT_ACTIVE_MS, activeMilliseconds, operationFailure, operationTimeout, type Operation } from './execution.js'; @@ -63,7 +62,6 @@ function controls(working: boolean): void { versionPicker.disabled = busy; hashInput.disabled = busy; el<HTMLButtonElement>('load-release').disabled = busy || !programPackage(programPicker.value); el<HTMLButtonElement>('load').disabled = busy; - recoveryButtons(); runButton.disabled = busy || (draft ? !compiled : !loaded); runButton.textContent = draft ? 'Run draft' : 'Run published'; el('draft-run-note').hidden = !draft || compiled; @@ -358,64 +356,6 @@ downloadButton.onclick = () => { if (worker) { operation = 'downloading'; controls(true); deadline.reset(PACKAGE_ACTIVE_MS); clock(); worker.postMessage({ type: 'download' }); } else { downloadAfterLoad = true; newWorker(false, true); } }; -function recoveryButtons(): void { - try { - const saved = readDraft(localStorage); - el('recover-draft').hidden = !saved; el('forget-draft').hidden = !saved; el('download-draft-source').hidden = !saved; - el<HTMLButtonElement>('recover-draft').disabled = busy || (!!saved?.demo && entries.length === 0); - if (saved && !el('draft-save-status').textContent) el('draft-save-status').textContent = 'A saved draft is available on this device. Recover it to continue editing; recovery does not run code.'; - } catch { - el('recover-draft').hidden = true; el('download-draft-source').hidden = true; el('forget-draft').hidden = false; - el('draft-save-status').textContent = 'Draft recovery is unavailable. Keep a separate copy of your source. You can try Forget saved draft to remove invalid saved data.'; - } -} -function saveDraft(): boolean { - if (!draft) return true; - try { - writeDraft(localStorage, { schema: 1, source: sourceEditor.value, name: exportsInput.value || preferredName, - arguments: argsInput.value, output: resultCodec.value, mode: mode.value === 'io' ? 'io' : 'pure', - ...(!scratch && selected?.environment !== undefined ? { demo: { hash: selected.hash, environment: selected.environment } } : {}) }); - el('draft-save-status').textContent = 'Draft saved on this device for recovery (latest draft only). Browser storage may be cleared; keep important work elsewhere too.'; - recoveryButtons(); return true; - } catch { - el('draft-save-status').closest<HTMLDetailsElement>('details')!.open = true; - el('draft-save-status').textContent = 'Draft recovery copy could not be saved. You can still compile; keep a separate copy of your source.'; - return false; - } -} -el<HTMLButtonElement>('recover-draft').onclick = () => { - try { - const saved = readDraft(localStorage); if (!saved) return; - const entry = saved.demo ? entries.find(e => e.hash === saved.demo!.hash && e.environment === saved.demo!.environment) : undefined; - if (saved.demo && !entry) throw new Error('The saved demo environment is no longer in this catalogue. Use Download saved source to recover your text; it has not been compiled against a substitute environment.'); - const output = outputType(saved.output); - dispose(); clearOutput(); clearTreeExport(); - chooseProgram(entry ? `demo:${entry.id}` : '', entry ? releases.find(r => r.demo === entry.id && r.hash === entry.hash)?.version : undefined); - editingSource = true; el<HTMLDetailsElement>('source-section').open = true; - draft = true; scratch = !saved.demo; selected = entry; originalSource = undefined; selection = undefined; loaded = false; - showRelease(entry ? releases.find(r => r.hash === entry.hash) : undefined); hashInput.value = entry?.hash ?? ''; - el('package-info').textContent = entry ? `Recovered draft based on ${entry.title}. No published executable loaded.` : 'Recovered scratch program. No published package selected.'; - sourceEditor.value = saved.source; sourceEditor.hidden = false; el('source-notes').hidden = true; - preferredName = saved.name; exportsInput.replaceChildren(); argsInput.value = saved.arguments; argumentsUI.refresh(true); resultCodec.value = output; mode.value = saved.mode; - // Recovery uses the page default, never stored resource settings. - const profile = el<HTMLInputElement>('compiler-uncapped'); profile.checked = true; profile.dispatchEvent(new Event('change')); - el('source-label').textContent = entry ? `Recovered draft · ${entry.title}` : 'Recovered scratch program · contract helpers'; - controls(false); el('status').textContent = 'Recovered draft: not compiled'; - } catch (error) { el('draft-save-status').textContent = error instanceof Error ? error.message : String(error); } -}; -el<HTMLButtonElement>('download-draft-source').onclick = () => { - try { - const saved = readDraft(localStorage); if (!saved) return; - const url = URL.createObjectURL(new Blob([saved.source], { type: 'text/plain;charset=utf-8' })); - const anchor = document.createElement('a'); anchor.href = url; anchor.download = 'local-draft.tri'; anchor.click(); - setTimeout(() => URL.revokeObjectURL(url), 1000); - } catch { el('draft-save-status').textContent = 'Saved source is unavailable.'; } -}; -el<HTMLButtonElement>('forget-draft').onclick = () => { - try { localStorage.removeItem(DRAFT_KEY); el('draft-save-status').textContent = 'Saved recovery copy removed. Editor contents are unchanged.'; recoveryButtons(); } - catch { el('draft-save-status').textContent = 'Unable to remove saved draft from local storage.'; } -}; -recoveryButtons(); el<HTMLButtonElement>('edit-source').onclick = () => { editingSource = true; el<HTMLDetailsElement>('source-section').open = true; controls(false); sourceEditor.focus(); }; @@ -426,7 +366,6 @@ sourceEditor.oninput = () => { dispose(); clearOutput(); clearTreeExport(); exportsInput.replaceChildren(); draft = scratch || sourceEditor.value !== originalSource; controls(false); - if (draft) saveDraft(); if (!draft && selection) { preferredExport = preferredName; newWorker(false, true); } else el('status').textContent = 'Ready to compile'; }; @@ -441,7 +380,6 @@ function compileDraft(): void { dispose(); operation = 'compiling'; clearOutput(); clearTreeExport(); exportsInput.replaceChildren(); try { sourceText(sourceEditor.value); - saveDraft(); settings = readSettings('compiler'); const current = new Worker(new URL('./compiler-worker.js', import.meta.url), { type: 'module' }); worker = current; controls(true); deadline.reset(activeMilliseconds(settings)); clock(); el('status').textContent = 'Compiling'; @@ -506,7 +444,7 @@ for (const button of document.querySelectorAll<HTMLButtonElement>('[data-compile exportsInput.replaceChildren(); preferredName = example.name; sourceEditor.hidden = false; el('source-notes').hidden = true; sourceEditor.value = example.source; argsInput.value = JSON.stringify(example.args, null, 2); argumentsUI.refresh(true); resultCodec.value = example.output; mode.value = example.mode; - saveDraft(); controls(false); el('source-label').textContent = 'Scratch program · contract helpers'; el('status').textContent = 'Ready to compile'; + controls(false); el('source-label').textContent = 'Scratch program · contract helpers'; el('status').textContent = 'Ready to compile'; }; function loadFragment(): boolean { const fragment = new URLSearchParams(location.hash.slice(1)); @@ -550,7 +488,7 @@ async function initializePackages(): Promise<void> { }); const indexBytes = await fetchBytes(new URL('../packages.json', import.meta.url), 1_048_576); releases = packageIndex(JSON.parse(new TextDecoder('utf-8', { fatal: true }).decode(indexBytes)) as unknown); - populatePrograms(); recoveryButtons(); + populatePrograms(); if (!packagePane() || selection || draft) { operation = undefined; return; } if (!loadFragment()) { const initial = entries.find(e => e.id === 'intro') ?? entries[0]; if (!initial) throw new Error('Empty catalogue'); select(initial); } } catch (error) { initialized = false; loaded = false; finish(error instanceof Error ? error.message : String(error), true); } diff --git a/ext/demo-site/src/bundle.ts b/ext/demo-site/src/bundle.ts @@ -5,9 +5,9 @@ type RecordNode = { kind: 'leaf' } | { kind: 'stem'; child: number } | { kind: ' export interface Bundle { exports: ExportInfo[]; roots: number[]; nodes: RecordNode[]; metadata: Record<string, string>; } -export const BUNDLE_LIMIT = 1_048_576; +export const BUNDLE_LIMIT = 16_777_216; // Separate bounded profile for the explicitly selected compiler, not packages. -export const COMPILER_BUNDLE_LIMIT = 2 * BUNDLE_LIMIT; +export const COMPILER_BUNDLE_LIMIT = 2_097_152; const utf8 = new TextDecoder('utf-8', { fatal: true }); class Reader { private at = 0; diff --git a/ext/demo-site/src/compiler-worker.ts b/ext/demo-site/src/compiler-worker.ts @@ -2,10 +2,10 @@ import { environmentProfile } from './packages.js'; import { executionSettings, executionBudget, collectionPolicy, type ExecutionSettings } from './execution.js'; import { contractEnvironment } from './contracts.js'; import { Graph, type NodeId } from './graph.js'; -import { COMPILER_BUNDLE_LIMIT, bundleHash, fetchBundle, fetchBytes, loadBundle } from './bundle.js'; +import { COMPILER_BUNDLE_LIMIT, fetchBundle, fetchBytes, loadBundle } from './bundle.js'; import { literals, makeLiteral, outputType, view, list, text } from './codecs.js'; import { runIO } from './io.js'; -import { COMPILER_EXPORT, sourceLiteral, compiledDefinitions } from './compiler.js'; +import { COMPILER_EXPORT, compilerDescriptor, sourceLiteral, compiledDefinitions } from './compiler.js'; import type { CompilerClient, CompilerMessage } from './compiler-protocol.js'; import type { RequestId } from './protocol.js'; const scope = globalThis as unknown as { postMessage(message: CompilerMessage): void; onmessage: ((event: MessageEvent<unknown>) => void) | null }; @@ -28,9 +28,7 @@ async function compile(source: unknown, settings: ExecutionSettings, profileName const input = sourceLiteral(graph, source); // Validate before downloading a compiler. const bytes = await fetchBytes(new URL('compiler.json', base), 4096); const raw: unknown = JSON.parse(new TextDecoder('utf-8', { fatal: true }).decode(bytes)); - if (!raw || typeof raw !== 'object') throw new Error('Invalid compiler descriptor'); - const descriptor = raw as Record<string, unknown>, hash = bundleHash(descriptor['hash']); - if (descriptor['schema'] !== 1 || descriptor['backend'] !== 'stable' || descriptor['export'] !== COMPILER_EXPORT) throw new Error('Invalid stable compiler descriptor'); + const { hash } = compilerDescriptor(raw); const bundle = await fetchBundle(new URL(`/_arboricx/bundle/hash/${hash}`, base), hash, COMPILER_BUNDLE_LIMIT); const library = loadBundle(graph, bundle), compiler = library.get(COMPILER_EXPORT); if (compiler === undefined) throw new Error('Missing stable compiler entrypoint'); diff --git a/ext/demo-site/src/compiler.ts b/ext/demo-site/src/compiler.ts @@ -1,6 +1,13 @@ +import { bundleHash, type BundleHash } from './bundle.js'; import { Graph, type NodeId } from './graph.js'; import { text, readText, readList, readNatural } from './codecs.js'; export const COMPILER_EXPORT = 'source_unit.compileStableModuleDefinitions'; +export function compilerDescriptor(raw: unknown): { hash: BundleHash } { + if (!raw || typeof raw !== 'object' || Array.isArray(raw)) throw new Error('Invalid compiler descriptor'); + const descriptor = raw as Record<string, unknown>; + if (descriptor['schema'] !== 1 || descriptor['backend'] !== 'stable' || descriptor['export'] !== COMPILER_EXPORT) throw new Error('Invalid stable compiler descriptor'); + return { hash: bundleHash(descriptor['hash']) }; +} export const COMPILER_BUDGET = Object.freeze({ nodes: 6_000_000, fuel: Infinity, stack: 100_000 }); export const COMPILER_COLLECTION = Object.freeze({ threshold: 250_000 }); export const SOURCE_LIMIT = 16_384; diff --git a/ext/demo-site/src/demo-inputs.ts b/ext/demo-site/src/demo-inputs.ts @@ -8,6 +8,7 @@ export function demoInputLabels(id: string, definition: string): readonly string case 'shipping': return ['Destination', 'Cart subtotal (cents)', 'Package weight (grams)']; case 'events': return definition === 'vendorB' ? ['Event [customer refunded totalCents]'] : ['Event [eventName customer totalCents]']; case 'registration': return ['Name', 'Email', 'Age', 'Terms accepted (t = false, (t t) = true)']; + case 'lexer': return definition === 'scanWhitespace' ? ['Text with leading whitespace'] : ['Text with a lowercase prefix']; case 'parser': return ['Comma-separated lowercase names']; case 'migration': return ['Profile [name email]']; default: return []; diff --git a/ext/demo-site/src/draft-storage.ts b/ext/demo-site/src/draft-storage.ts @@ -1,39 +0,0 @@ -// One local recovery slot, never executable state or an automatic run request. -// Browser limits are intentionally NOT persisted: recovery starts in Standard. -export const DRAFT_KEY = 'arboricx.local-draft.v1'; -export interface SavedDraft { - schema: 1; - source: string; - name: string; - arguments: string; - output: string; - mode: 'pure' | 'io'; - demo?: { hash: string; environment: string }; -} -export function savedDraft(raw: unknown): SavedDraft { - if (!raw || typeof raw !== 'object' || Array.isArray(raw)) throw new Error('Invalid saved draft'); - const v = raw as Record<string, unknown>; - for (const [key, max] of [['source', 32768], ['name', 256], ['arguments', 16000], ['output', 32]] as const) - if (typeof v[key] !== 'string' || v[key].length > max) throw new Error('Invalid saved draft ' + key); - if (v['schema'] !== 1 || (v['mode'] !== 'pure' && v['mode'] !== 'io')) throw new Error('Invalid saved draft version/mode'); - let demo: SavedDraft['demo']; - if (v['demo'] !== undefined) { - const d = v['demo'] as Record<string, unknown>; - if (!d || typeof d !== 'object' || typeof d['hash'] !== 'string' || !/^[0-9a-f]{64}$/u.test(d['hash']) || typeof d['environment'] !== 'string' || !/^[A-Za-z0-9_-]{1,128}$/u.test(d['environment'])) throw new Error('Invalid saved demo context'); - demo = { hash: d['hash'], environment: d['environment'] }; - } - return { schema: 1, source: v['source'] as string, name: v['name'] as string, arguments: v['arguments'] as string, - output: v['output'] as string, mode: v['mode'], ...(demo ? { demo } : {}) }; -} -export function readDraft(storage: Pick<Storage, 'getItem'>): SavedDraft | undefined { - const text = storage.getItem(DRAFT_KEY); - if (text === null) return undefined; - if (text.length > 100000) throw new Error('Saved draft too large'); - return savedDraft(JSON.parse(text) as unknown); -} -export function writeDraft(storage: Pick<Storage, 'setItem' | 'getItem'>, draft: SavedDraft): void { - const text = JSON.stringify(savedDraft(draft)); - if (text.length > 100000) throw new Error('Saved draft too large'); - storage.setItem(DRAFT_KEY, text); - if (storage.getItem(DRAFT_KEY) !== text) throw new Error('Draft storage verification failed'); -} diff --git a/ext/demo-site/style.css b/ext/demo-site/style.css @@ -185,7 +185,6 @@ button.primary.draft { background:#8b581b; } .source-panel details > summary { color:#c6dacb; } .resource-choice { border:1px solid var(--line); border-radius:8px; padding:12px 16px; margin:12px 0; } .resource-choice p { margin:4px 0 8px; } -.draft-recovery { padding:0 20px; } /* A single primary path, with secondary controls disclosed on demand. */ .workspace-tabs { display:grid; grid-template-columns:repeat(3,minmax(0,1fr)); } @@ -210,8 +209,6 @@ button.primary.draft { background:#8b581b; } .run-tools { margin-top:16px; margin-bottom:18px; border-top:1px solid var(--line); padding-top:12px; } .run-tools .execution-limits { margin:12px 0; } .trace-panel > summary { padding:14px 20px; } -.draft-recovery { margin:10px 0; } -.draft-recovery .hint:empty { display:none; } @media(max-width:540px) { .workspace-tabs { grid-template-columns:repeat(3,minmax(0,1fr)); } .workspace-tabs button { font-size:12px; padding:10px 2px; } diff --git a/ext/demo-site/test/browser.mjs b/ext/demo-site/test/browser.mjs @@ -9,7 +9,7 @@ import { tmpdir } from 'node:os'; import { resolve, extname, sep } from 'node:path'; import { fileURLToPath } from 'node:url'; import { createHash } from 'node:crypto'; -import { makeBundle } from './bundle-fixture.mjs'; +import { makeBundle, sizedBundle } from './bundle-fixture.mjs'; const root = resolve(process.env['ARBORICX_SITE_ROOT'] || fileURLToPath(new URL('../', import.meta.url))); const dependencyBytes = hash => readFile(process.env['ARBORICX_REGISTRY_STORE'] @@ -19,6 +19,7 @@ const prefix = '/nested/tricu/'; // Exercise static subdirectory deployment. const registry = new Map(); const pin = bytes => createHash('sha256').update(bytes).digest('hex'); const published = {}; +for(const [name,size] of [['boundary',16_777_216],['oversized',16_777_217]]){const bytes=sizedBundle(size);published[name]=pin(bytes);registry.set(published[name],bytes);} for(const name of ['id','append','size']) { const bytes=await readFile(new URL(`../../../test/fixtures/${name}.arboricx`,import.meta.url)); published[name]=pin(bytes); registry.set(published[name],bytes); @@ -66,7 +67,7 @@ const server = createServer(async (request, response) => { if(pathname.startsWith(prefix+'bundles/'))throw new Error('Bundles are not static assets'); if(pathname.startsWith('/_arboricx/bundle/hash/')) { const hash=pathname.slice('/_arboricx/bundle/hash/'.length); - if(hash==='a'.repeat(64)){response.setHeader('Content-Length','1048577');response.end();return;} + if(hash==='a'.repeat(64)){response.setHeader('Content-Length','16777217');response.end();return;} const bytes=hash==='f'.repeat(64)?registry.get(published.id):registry.get(hash); if(!bytes)throw new Error('missing bundle'); response.setHeader('Content-Type','application/vnd.arboricx.bundle');response.end(bytes);return; @@ -348,6 +349,50 @@ try { await click('run');await wait(status('Stopped')); assert.match(await evaluate("document.getElementById('result').textContent"),/^Program execution failed: Invalid canonical number/,'native action exports are not standalone browser tricuIO programs'); console.log('PASS parser demo / complete vs prefix parsing / committed failures / read-only source / native server action is not executable as browser IO'); + await evaluate("document.getElementById('program-picker').value='demo:lexer';document.getElementById('program-picker').dispatchEvent(new Event('change'))");await wait(status('Ready')); + assert.equal(await evaluate("document.getElementById('source-summary').textContent"),'Source (read-only)'); + assert.equal(await evaluate("document.getElementById('compile').hidden"),true); + await click('run');await wait(status('Finished')); + assert.deepEqual(await resultRecord(),{matched:'hello',remaining:',world',line:1,column:6,offset:5,consumed:true}); + await evaluate("document.getElementById('export').value='scanWhitespace';document.getElementById('export').dispatchEvent(new Event('change'))"); + assert.deepEqual(await evaluate("[...document.querySelectorAll('#argument-fields label')].map(l=>l.textContent)"),['Text with leading whitespace']); + await click('run');await wait(status('Finished')); + assert.deepEqual(await resultRecord(),{matched:' \n\t',remaining:'word',line:2,column:2,offset:3,consumed:true}); + await evaluate("document.getElementById('program-picker').value='package:tricu-math';document.getElementById('program-picker').dispatchEvent(new Event('change'))"); + await click('load-release');await wait(status('Ready')); + await evaluate("document.getElementById('export').value='add';document.getElementById('export').dispatchEvent(new Event('change'));document.getElementById('arguments').value=JSON.stringify([{type:'number',value:'123456789'},{type:'number',value:'11'}]);document.getElementById('output-type').value='number'"); + await click('run');await wait(status('Finished')); + assert.equal(await evaluate("document.getElementById('result').textContent"),'Returned number: "123456800"'); + await evaluate("document.getElementById('program-picker').value='package:tricu-compiler';document.getElementById('program-picker').dispatchEvent(new Event('change'))"); + await click('load-release');await wait(status('Ready')); + assert.equal(await evaluate("document.getElementById('source-summary').textContent"),'Package details'); + assert.equal(await evaluate("document.getElementById('edit-source').hidden"),true); + assert.equal(await evaluate("document.getElementById('export').options.length"),1169); + assert.match(await evaluate("document.getElementById('package-info').textContent"),new RegExp(compilerDescriptor.hash)); + const indexedCompilerRequests=requests.filter(p=>p.endsWith('/dist/compiler-worker.js')).length; + await click('tab-export-definition'); + await evaluate("document.getElementById('tree-definition').value='source_unit.compileStableModuleDefinitions';document.getElementById('tree-definition').dispatchEvent(new Event('change'));document.getElementById('tree-format').value='dag';document.getElementById('tree-format').dispatchEvent(new Event('change'))"); + await click('export-tree');await wait("!document.getElementById('copy-tree').disabled"); + assert.match(await evaluate("document.getElementById('tree-provenance').textContent"),/source_unit.compileStableModuleDefinitions/); + assert.match(await evaluate("document.getElementById('tree-provenance').textContent"),new RegExp(compilerDescriptor.hash)); + assert.equal(requests.filter(p=>p.endsWith('/dist/compiler-worker.js')).length,indexedCompilerRequests,'loading/exporting the compiler library never creates a draft compiler Worker'); + await click('tab-playground'); + // Hash loading/sharing must recognize the indexed compiler too, not only its name. + await evaluate(`document.getElementById('program-picker').value='hash';document.getElementById('program-picker').dispatchEvent(new Event('change'));document.getElementById('bundle-hash').value=${JSON.stringify(compilerDescriptor.hash)};document.getElementById('program-hash').dispatchEvent(new Event('submit',{cancelable:true}))`);await wait(status('Ready')); + assert.equal(await evaluate("document.getElementById('export').options.length"),1169); + await evaluate("document.getElementById('program-picker').value='package:tricu-source-builder';document.getElementById('program-picker').dispatchEvent(new Event('change'))"); + await click('load-release');await wait(status('Ready')); + assert.deepEqual(await evaluate("[...document.getElementById('export').options].map(o=>o.value)"),['sourceBuildGraph']); + assert.match(await evaluate("document.getElementById('release-info').textContent"),/native application compiler/); + for(const [name,expected] of [['boundary','Ready'],['oversized','Stopped']]){ + await evaluate(`document.getElementById('program-picker').value='hash';document.getElementById('program-picker').dispatchEvent(new Event('change'));document.getElementById('bundle-hash').value=${JSON.stringify(published[name])};document.getElementById('program-hash').dispatchEvent(new Event('submit',{cancelable:true}))`);await wait(status(expected)); + if(name==='boundary'){ + assert.equal(await evaluate("document.getElementById('export').options.length"),1); + assert.equal(await evaluate("document.getElementById('export').value"),'root'); + }else assert.match(await evaluate("document.getElementById('result').textContent"),/download limit/); + } + console.log('PASS 16 MiB package boundary / over-limit rejection with optional execution guards off'); + console.log('PASS lexer positions / pure arithmetic / indexed compiler exact pin and DAG export / compiler hash loading / native builder identity'); for(const [demo,exports] of [['shipping',['v1','v2']],['events',['vendorA','vendorB']],['registration',['main']],['migration',['v1','v2']]]){ await evaluate(`document.getElementById('program-picker').value='demo:${demo}';document.getElementById('program-picker').dispatchEvent(new Event('change'))`);await wait(status('Ready')); assert.equal(await evaluate("document.getElementById('source-section').open"),true,'every selected demo starts with readable source'); @@ -996,53 +1041,46 @@ try { await writeFile(process.argv[3].replace(/\.png$/,'')+'-export-'+width+'.png',Buffer.from(shot.data,'base64')); } } - await click('tab-playground'); - await evaluate("document.querySelector('[data-compiler-example=identity]').click();document.getElementById('source-editor').value='id = (x : x)\\n-- recovery test';document.getElementById('source-editor').dispatchEvent(new Event('input'));document.getElementById('compiler-uncapped').checked=true;document.getElementById('compiler-uncapped').dispatchEvent(new Event('change'))"); - await click('compile');await wait(compilerStatus('Compiled')); - const savedSource=await evaluate("JSON.parse(localStorage.getItem('arboricx.local-draft.v1')).source"); - async function reloadForRecovery(){ - const origin=await evaluate('performance.timeOrigin'); - await send('Page.reload'); - await wait(`performance.timeOrigin!==${origin} && document.querySelector('#package-settings .limits-editor')!==null && document.getElementById('recover-draft')?.hidden===false`); - } - - const compilerRequests=requests.filter(path=>path.endsWith('/compiler.json')).length; - await reloadForRecovery(); - await click('tab-playground');await wait(status('Ready')); - assert.equal(requests.filter(path=>path.endsWith('/compiler.json')).length,compilerRequests,'reload does not compile'); - assert.equal(await evaluate("document.getElementById('compiler-uncapped').checked"),true); - await click('recover-draft'); - assert.equal(await evaluate("document.getElementById('source-editor').value"),savedSource); - assert.equal(await evaluate("document.getElementById('status').textContent"),'Recovered draft: not compiled'); - assert.equal(requests.filter(path=>path.endsWith('/compiler.json')).length,compilerRequests,'recovery does not compile'); + // Even blocked browser storage and an older saved draft must be irrelevant. + const storageScript=await send('Page.addScriptToEvaluateOnNewDocument',{source:` + localStorage.setItem('arboricx.local-draft.v1',JSON.stringify({schema:1,source:'oldDraft = 999',name:'oldDraft',arguments:'[]',output:'number',mode:'pure'})); + globalThis.storageTouches=[]; + for(const key of ['localStorage','sessionStorage'])Object.defineProperty(globalThis,key,{configurable:true,get(){globalThis.storageTouches.push(key);throw new DOMException('blocked','SecurityError');}}); + globalThis.confirm=()=>{throw Error('Unexpected confirmation');};`}); + await evaluate("history.replaceState(null,'',location.pathname)"); + await send('Page.reload');await openPlayground(); + const original=await evaluate("document.getElementById('source-editor').value"); + assert.doesNotMatch(original,/oldDraft/); + assert.equal(await evaluate("document.getElementById('program-picker').value"),'demo:intro'); + assert.equal(await evaluate("document.querySelector('.draft-recovery,#recover-draft,#forget-draft,#download-draft-source')"),null); + await evaluate("document.querySelector('[data-compiler-example=identity]').click();document.getElementById('source-editor').value='id = (x : x)\\n-- page-local draft';document.getElementById('source-editor').dispatchEvent(new Event('input'))"); await click('compile');await wait(compilerStatus('Compiled')); assert.match(await evaluate("document.getElementById('compiler-stats').textContent"),/optional limits off/); - // Recovery failure is reported inline, but does not gate explicit compilation. - await evaluate("window._setItem=Storage.prototype.setItem;Storage.prototype.setItem=()=>{throw new DOMException('blocked','QuotaExceededError')};window.confirm=()=>{throw Error('Unexpected confirmation')};document.getElementById('source-editor').value+='\\n-- not saved';document.getElementById('source-editor').dispatchEvent(new Event('input'))"); - await click('compile');await wait(compilerStatus('Compiled')); - assert.match(await evaluate("document.getElementById('draft-save-status').textContent"),/could not be saved/); - assert.equal(await evaluate("document.getElementById('draft-save-status').closest('details').open"),true); - assert.equal(await evaluate("JSON.parse(localStorage.getItem('arboricx.local-draft.v1')).source"),savedSource,'failed save does not replace the previous recovery copy'); - await evaluate("Storage.prototype.setItem=window._setItem"); - await click('forget-draft'); - assert.equal(await evaluate("localStorage.getItem('arboricx.local-draft.v1')"),null); - assert.equal(await evaluate("document.getElementById('recover-draft').hidden"),true); - assert.match(await evaluate("document.getElementById('source-editor').value"),/not saved/); - // Curated recovery must keep its environment, never become a scratch program. - await evaluate("document.getElementById('program-picker').value='demo:shipping';document.getElementById('program-picker').dispatchEvent(new Event('change'))");await wait(status('Ready')); - await evaluate("document.getElementById('source-editor').value='answer = stringField \"status\" \"recovered\"';document.getElementById('arguments').value='[]';document.getElementById('output-type').value='list';document.getElementById('source-editor').dispatchEvent(new Event('input'))"); - assert.equal(await evaluate("JSON.parse(localStorage.getItem('arboricx.local-draft.v1')).demo.environment"),'practical'); - await reloadForRecovery(); - await click('tab-playground');await wait(status('Ready')); - await click('recover-draft');await click('compile');await wait(status('Compiled'));await click('run');await wait(status('Finished')); - assert.match(await evaluate("document.getElementById('result').textContent"),/^Returned list:/); - await evaluate("const d=JSON.parse(localStorage.getItem('arboricx.local-draft.v1'));d.demo.hash='0'.repeat(64);localStorage.setItem('arboricx.local-draft.v1',JSON.stringify(d))"); - const beforeMissingContext=requests.filter(path=>path.endsWith('/compiler.json')).length; - await click('recover-draft'); - assert.match(await evaluate("document.getElementById('draft-save-status').textContent"),/no longer in this catalogue/); - assert.equal(requests.filter(path=>path.endsWith('/compiler.json')).length,beforeMissingContext); - await click('forget-draft'); - console.log('PASS direct compilation / inline memory notice / saved source before compile / reload recovery with default guards off / no auto-execution / storage failure / curated environment / missing-context rejection / forget'); + const liveSource=await evaluate("document.getElementById('source-editor').value"); + await click('run');await wait(status('Finished')); + assert.equal(await evaluate("document.getElementById('result').textContent"),'Returned number: "42"'); + await click('stop-compile'); + assert.equal(await evaluate("document.getElementById('source-editor').value"),liveSource); + await evaluate("document.getElementById('source-editor').value='id = (';document.getElementById('source-editor').dispatchEvent(new Event('input'))"); + await click('compile');await wait(compilerStatus('Stopped')); + assert.equal(await evaluate("document.getElementById('source-editor').value"),'id = ('); + assert.deepEqual(await evaluate('globalThis.storageTouches'),[]); + const beforeReload=requests.filter(path=>path.endsWith('/compiler.json')).length; + await send('Page.reload');await openPlayground(); + assert.equal(await evaluate("document.getElementById('source-editor').value"),original,'reload starts with published source, not the failed or saved draft'); + assert.equal(requests.filter(path=>path.endsWith('/compiler.json')).length,beforeReload,'reload does not compile'); + assert.equal(await evaluate("document.getElementById('run').textContent"),'Run published'); + assert.equal(await evaluate("document.getElementById('result').textContent"),''); + assert.deepEqual(await evaluate('globalThis.storageTouches'),[]); + await evaluate(`history.replaceState(null,'',location.pathname+'#bundle=${published.greet}&demo=greet&export=main')`); + await send('Page.reload');await openPlayground(); + assert.equal(await evaluate("document.getElementById('program-picker').value"),'demo:greet','a shared published selection wins over obsolete stored drafts'); + assert.equal(await evaluate("document.getElementById('source-editor').value"),greeting.source); + assert.equal(await evaluate("document.getElementById('terminal').textContent"),''); + assert.equal(requests.filter(path=>path.endsWith('/compiler.json')).length,beforeReload); + assert.deepEqual(await evaluate('globalThis.storageTouches'),[]); + await send('Page.removeScriptToEvaluateOnNewDocument',{identifier:storageScript.identifier}); + console.log('PASS page-local drafts / no storage access or recovery UI / blocked storage / old drafts ignored / source retained after Stop and errors / reload starts fresh without execution'); assert.deepEqual(exceptions, []); } finally { diff --git a/ext/demo-site/test/bundle-fixture.mjs b/ext/demo-site/test/bundle-fixture.mjs @@ -15,3 +15,17 @@ export function makeBundle(records, namedRoots=[['root',records.length-1]], opti const section=(type,offset,length)=>Buffer.concat([u32(type),u16(1),u16(1),u32(0),u64(offset),u64(length),u32(0)]); return Buffer.concat([Buffer.from('ARBORICX'),u16(1),u16(0),u32(2),u64(0),u64(32),section(1,96,manifest.length),section(2,96+manifest.length,nodes.length),manifest,nodes]); } +// A valid tiny DAG plus one opaque, noncritical section: byte-boundary controls +// without relaxing node limits or manufacturing enormous executable trees. +export function sizedBundle(size){ + const small=makeBundle([[]]); + if(size<small.length+32)throw Error('Sized fixture too small'); + const out=Buffer.alloc(size); + small.copy(out,0,0,96);out.writeUInt32BE(3,12); + for(let i=0;i<2;i++)out.writeBigUInt64BE(small.readBigUInt64BE(44+32*i)+32n,44+32*i); + out.writeUInt32BE(100,96);out.writeUInt16BE(1,100); + out.writeBigUInt64BE(BigInt(small.length+32),108); + out.writeBigUInt64BE(BigInt(size-small.length-32),116); + small.copy(out,128,96); + return out; +} diff --git a/ext/demo-site/test/bundle.test.mjs b/ext/demo-site/test/bundle.test.mjs @@ -4,7 +4,7 @@ import {readFile} from 'node:fs/promises'; import {createHash} from 'node:crypto'; import {Graph} from '../dist/graph.js'; import {COMPILER_BUNDLE_LIMIT,decodeBundle,loadBundle,bundleHash,fetchBytes,fetchBundle,BUNDLE_LIMIT} from '../dist/bundle.js'; -import {makeBundle} from './bundle-fixture.mjs'; +import {makeBundle,sizedBundle} from './bundle-fixture.mjs'; // Audited publication: 1152 prior exports + 5 IO transport helpers, // sourceObjectFlatTreeDomain, and 11 pattern-dispatch helpers; no removals. const COMPILER_EXPORT_COUNT=1169; @@ -21,7 +21,7 @@ test('stable packaged sources/pins and real compiler bundle decode',async()=>{ const descriptor=JSON.parse(await readFile(new URL('../compiler.json',import.meta.url))); const bytes=await readFile(new URL('../bundles/'+descriptor.hash+'.arboricx',import.meta.url)); assert.equal(createHash('sha256').update(bytes).digest('hex'),data.compilerBundleSha256); - assert.throws(()=>decodeBundle(bytes),/byte limit/); + assert.equal(decodeBundle(bytes).exports.length,COMPILER_EXPORT_COUNT); const bundle=decodeBundle(bytes,COMPILER_BUNDLE_LIMIT);assert.equal(bundle.exports.length,COMPILER_EXPORT_COUNT); const g=new Graph();loadBundle(g,bundle);assert.equal(g.stats.nodes,bundle.nodes.length); }); @@ -100,8 +100,22 @@ test('compiler download profile retains header/stream cancellation and hash veri const descriptor=JSON.parse(await readFile(new URL('../compiler.json',import.meta.url))); const bytes=await readFile(new URL('../bundles/'+descriptor.hash+'.arboricx',import.meta.url)); globalThis.fetch=async()=>new Response(bytes); - await assert.rejects(fetchBundle(url,descriptor.hash),/download limit/); + assert.equal((await fetchBundle(url,descriptor.hash)).exports.length,COMPILER_EXPORT_COUNT); assert.equal((await fetchBundle(url,descriptor.hash,COMPILER_BUNDLE_LIMIT)).exports.length,COMPILER_EXPORT_COUNT); await assert.rejects(fetchBundle(url,await sha256(new Uint8Array()),COMPILER_BUNDLE_LIMIT),/SHA-256/); }finally{globalThis.fetch=saved;} }); + + +test('ordinary packages accept exactly 16 MiB while draft compiler input stays at 2 MiB',async()=>{ + assert.equal(BUNDLE_LIMIT,16_777_216);assert.equal(COMPILER_BUNDLE_LIMIT,2_097_152); + const bytes=sizedBundle(BUNDLE_LIMIT),hash=bundleHash(createHash('sha256').update(bytes).digest('hex')); + assert.equal(decodeBundle(bytes).nodes.length,1); + assert.throws(()=>decodeBundle(bytes,COMPILER_BUNDLE_LIMIT),/byte limit/); + assert.throws(()=>decodeBundle(sizedBundle(BUNDLE_LIMIT+1)),/byte limit/); + const saved=globalThis.fetch; + try{ + globalThis.fetch=async()=>new Response(bytes); + assert.equal((await fetchBundle(new URL('../compiler.json',import.meta.url),hash)).nodes.length,1); + }finally{globalThis.fetch=saved;} +}); diff --git a/ext/demo-site/test/compiler-package.test.mjs b/ext/demo-site/test/compiler-package.test.mjs @@ -0,0 +1,55 @@ +import test from 'node:test'; +import assert from 'node:assert/strict'; +import {readFile} from 'node:fs/promises'; +import {compilerDescriptor,COMPILER_EXPORT} from '../dist/compiler.js'; +import {BUNDLE_LIMIT} from '../dist/bundle.js'; +const descriptor=JSON.parse(await readFile(new URL('../compiler.json',import.meta.url))); +const bytes=await readFile(new URL('../bundles/'+descriptor.hash+'.arboricx',import.meta.url)); +let instance=0; +async function session(message,fetcher,after){ + const saved={fetch:globalThis.fetch,onmessage:globalThis.onmessage,postMessage:globalThis.postMessage}; + let timer; + try{ + globalThis.fetch=fetcher; + const messages=[]; + const reply=new Promise((resolve,reject)=>{ + timer=setTimeout(()=>reject(Error('worker test timeout')),5000); + globalThis.postMessage=data=>{messages.push(data);if(data.type==='loaded'||data.type==='error')resolve(data);}; + }); + await import('../dist/worker.js?test='+ ++instance); + globalThis.onmessage({data:message}); + const result=await reply; + if(after)await after(result,messages); + return result; + }finally{ + clearTimeout(timer); + for(const [key,value] of Object.entries(saved))if(value===undefined)delete globalThis[key];else globalThis[key]=value; + } +} + +test('stable compiler descriptors reject arrays, wrong backends/entrypoints and malformed pins',()=>{ + assert.equal(compilerDescriptor(descriptor).hash,descriptor.hash); + for(const raw of [null,[],{...descriptor,schema:2},{...descriptor,backend:'experimental'},{...descriptor,export:'main'},{...descriptor,hash:'A'.repeat(64)}])assert.throws(()=>compilerDescriptor(raw)); +}); + +test('the indexed compiler loads and downloads exact bytes as an ordinary package without compiling',async()=>{ + const requests=[]; + const fetcher=async url=>{requests.push(String(url));return new Response(bytes);}; + const loaded=await session({type:'load',hash:descriptor.hash},fetcher,async(result,messages)=>{ + assert.equal(result.type,'loaded');assert.ok(result.exports.some(e=>e.name===COMPILER_EXPORT)); + assert.equal(result.source,null);assert.ok(result.nodes<=100000); + globalThis.onmessage({data:{type:'download'}}); + const download=messages.at(-1);assert.equal(download.type,'downloaded');assert.equal(download.hash,descriptor.hash); + assert.deepEqual(Buffer.from(download.bytes),bytes); + assert.equal(messages.some(m=>m.type==='done'),false,'load/download never execute a program'); + }); + assert.equal(loaded.type,'loaded');assert.equal(requests.length,1); + assert.equal(requests.some(url=>url.endsWith('/compiler.json')),false); +}); + +test('ordinary packages retain their byte ceiling and SHA-256 verification',async()=>{ + const wrong=await session({type:'load',hash:'0'.repeat(64)},async()=>new Response(bytes)); + assert.match(wrong.message,/SHA-256 mismatch/); + const oversized=await session({type:'load',hash:descriptor.hash},async()=>new Response(new Uint8Array(BUNDLE_LIMIT+1))); + assert.match(oversized.message,/download limit/); +}); diff --git a/ext/demo-site/test/draft-session.test.mjs b/ext/demo-site/test/draft-session.test.mjs @@ -0,0 +1,25 @@ +import test from 'node:test'; +import assert from 'node:assert/strict'; +import {readFile,readdir,access} from 'node:fs/promises'; + +test('browser code has no draft persistence or retired storage module',async()=>{ + for(const directory of ['src','dist']){ + const root=new URL('../'+directory+'/',import.meta.url); + for(const name of await readdir(root)){ + if(!/\.(?:ts|js)$/.test(name))continue; + assert.doesNotMatch(await readFile(new URL(name,root),'utf8'),/\b(?:localStorage|sessionStorage|indexedDB|saveDraft|readDraft|writeDraft|recoveryButtons)\b|draft-storage/,name); + } + } + for(const file of ['src/draft-storage.ts','dist/draft-storage.js']) + await assert.rejects(access(new URL('../'+file,import.meta.url)),{code:'ENOENT'}); +}); + +test('source remains editable without autosave or recovery controls',async()=>{ + const html=await readFile(new URL('../index.html',import.meta.url),'utf8'); + assert.doesNotMatch(html,/draft-recovery|draft-save-status|recover-draft|forget-draft|download-draft-source|Draft recovery/); + for(const id of ['source-editor','edit-source','compile','stop-compile','restore-source']) + assert.match(html,new RegExp(`id="${id}"`)); + const readme=await readFile(new URL('../README.md',import.meta.url),'utf8'); + assert.match(readme,/Drafts exist only in the open page/); + assert.match(readme,/page reload\s+starts fresh/); +}); diff --git a/ext/demo-site/test/draft-storage.test.mjs b/ext/demo-site/test/draft-storage.test.mjs @@ -1,21 +0,0 @@ -import test from 'node:test'; -import assert from 'node:assert/strict'; -import {DRAFT_KEY,savedDraft,readDraft,writeDraft} from '../dist/draft-storage.js'; -const draft={schema:1,source:'invalid unfinished source is still saved',name:'id',arguments:'[]',output:'tree',mode:'pure'}; -function storage(){const values=new Map();return {getItem:k=>values.get(k)??null,setItem:(k,v)=>values.set(k,v),removeItem:k=>values.delete(k)};} -test('recovery retains source and exact demo context but no risky mode or executable state',()=>{ - const s=storage();assert.equal(readDraft(s),undefined); - const demo={hash:'a'.repeat(64),environment:'practical'}; - writeDraft(s,{...draft,demo,uncapped:true,compiled:true}); - assert.deepEqual(readDraft(s),{...draft,demo}); - s.removeItem(DRAFT_KEY);assert.equal(readDraft(s),undefined); -}); -test('recovery rejects malformed, oversized or unavailable storage without silent success',()=>{ - for(const value of [null,[],{}, {...draft,schema:2},{...draft,source:'x'.repeat(32769)},{...draft,demo:{hash:'bad',environment:'empty'}}])assert.throws(()=>savedDraft(value)); - const s=storage();s.setItem(DRAFT_KEY,'{');assert.throws(()=>readDraft(s)); - s.setItem(DRAFT_KEY,' '.repeat(100001));assert.throws(()=>readDraft(s),/large/); - assert.throws(()=>writeDraft(s,{...draft,source:'\0'.repeat(32768)}),/large/); - assert.throws(()=>writeDraft({setItem(){throw new Error('quota');}},draft),/quota/); - assert.throws(()=>writeDraft({setItem(){},getItem(){return null;}},draft),/verification/); - assert.throws(()=>readDraft({getItem(){throw new Error('denied');}}),/denied/); -}); diff --git a/ext/demo-site/test/graph.test.mjs b/ext/demo-site/test/graph.test.mjs @@ -7,7 +7,7 @@ import { natural, readNatural, text, readText, line, literals, makeLiteral, view import { runIO } from '../dist/io.js'; import { decodeBundle, loadBundle } from '../dist/bundle.js'; const data = JSON.parse(await readFile(new URL('../demos.json', import.meta.url))); -data.demos.parser=JSON.parse(await readFile(new URL('../parser-demo.json',import.meta.url))); +for(const id of ['parser','lexer'])data.demos[id]=JSON.parse(await readFile(new URL('../'+id+'-demo.json',import.meta.url))); function host(input = '') { const state = { terminal: '', trace: [] }; return { state, before: async event => state.trace.push(event), after() {}, running() {}, diff --git a/ext/demo-site/test/packages.test.mjs b/ext/demo-site/test/packages.test.mjs @@ -79,7 +79,7 @@ test('showcase packages retain browser-bounded complete library roots and native const release=releases.find(r=>r.name===name);assert.ok(release);assert.equal(release.demo,undefined); const bytes=await read('bundles/'+release.hash+'.arboricx'); assert.equal(createHash('sha256').update(bytes).digest('hex'),release.hash); - assert.ok(bytes.length<=1048576); + assert.ok(bytes.length<=16777216); const bundle=decodeBundle(bytes);assert.ok(bundle.nodes.length<=100000); const exports=loadBundle(new Graph(),bundle); for(const name of required)assert.ok(exports.has(name),name); @@ -97,3 +97,46 @@ test('native parser demo source and goldens are authenticated without claiming a assert.ok(goldens.haskellCases.some(c=>c.record.status==='error'&&c.record.committed)); assert.ok(goldens.haskellCases.some(c=>c.export==='parsePrefix'&&c.record.status==='ok'&&c.record.remaining)); }); + +test('both indexed compiler packages retain their exact independently authenticated artifacts',async()=>{ + const releases=packageIndex(JSON.parse(await read('packages.json'))),descriptor=JSON.parse(await read('compiler.json')); + for(const [name,path,entrypoint] of [ + ['tricu-compiler','test/fixtures/compiler-library.arboricx','source_unit.compileStableModuleDefinitions'], + ['tricu-source-builder','bootstrap/build-stable.arboricx','sourceBuildGraph'], + ]){ + const release=releases.find(r=>r.name===name);assert.ok(release);assert.equal(release.demo,undefined); + const bytes=await read('bundles/'+release.hash+'.arboricx'); + assert.deepEqual(bytes,await readFile(new URL('../../../'+path,import.meta.url))); + assert.equal(createHash('sha256').update(bytes).digest('hex'),release.hash); + const {COMPILER_BUNDLE_LIMIT}=await import('../dist/bundle.js'); + const bundle=decodeBundle(bytes,COMPILER_BUNDLE_LIMIT);assert.ok(bundle.nodes.length<=100000); + assert.ok(bundle.exports.some(e=>e.name===entrypoint)); + if(name==='tricu-compiler')assert.equal(release.hash,descriptor.hash); + else{assert.equal(bundle.exports.length,1);assert.match(release.title,/native application compiler/);} + } +}); + +test('new practical libraries expose complete interfaces and lexer reference source is authenticated',async()=>{ + const releases=packageIndex(JSON.parse(await read('packages.json'))); + for(const [name,required] of Object.entries({ + 'tricu-math':['add','divMod','inRange','decimal','printable'], + 'tricu-lexer':['lexerScanWhile','lexerAdvance','lexerDispatch','lexerToken'], + 'tricu-binary':['readU8','readBytes','binaryUtf8?','errInvalidBinaryInput'], + 'tricu-list-traversal':['listContinue','listStop','listFoldUntil','listFoldTailUntil'], + 'tricu-patterns':['match','otherwise','cond','guard'], + })){ + const release=releases.find(r=>r.name===name);assert.ok(release);assert.equal(release.demo,undefined); + const bytes=await read('bundles/'+release.hash+'.arboricx'); + assert.equal(createHash('sha256').update(bytes).digest('hex'),release.hash); + const names=decodeBundle(bytes).exports.map(e=>e.name); + for(const name of required)assert.ok(names.includes(name),name); + if(name==='tricu-list-traversal'||name==='tricu-patterns')assert.deepEqual(names.sort(),required.sort()); + } + const demo=JSON.parse(await read('catalogue.json')).find(e=>e.id==='lexer'),goldens=JSON.parse(await read('lexer-demo.json')); + assert.equal(demo.preparation,'native');assert.equal(demo.environment,undefined); + assert.equal(demo.source,await read('demos/lexer.tri').then(b=>b.toString())); + assert.equal(createHash('sha256').update(demo.source).digest('hex'),demo.sourceSha256); + assert.equal(goldens.hash,demo.hash);assert.equal(goldens.sourceSha256,demo.sourceSha256); + assert.ok(goldens.haskellCases.some(c=>c.record.line===2&&c.record.offset===3)); + assert.ok(goldens.haskellCases.some(c=>c.record.remaining==='é')); +}); diff --git a/ext/demo-site/test/practical-packages.test.mjs b/ext/demo-site/test/practical-packages.test.mjs @@ -0,0 +1,44 @@ +import test from 'node:test'; +import assert from 'node:assert/strict'; +import {readFile} from 'node:fs/promises'; +import {Graph} from '../dist/graph.js'; +import {decodeBundle,loadBundle,COMPILER_BUNDLE_LIMIT} from '../dist/bundle.js'; +import {natural,readNatural,readList,view} from '../dist/codecs.js'; +const index=JSON.parse(await readFile(new URL('../packages.json',import.meta.url))).releases; +const bundle=async name=>decodeBundle(await readFile(new URL('../bundles/'+index.find(r=>r.name===name).hash+'.arboricx',import.meta.url)),COMPILER_BUNDLE_LIMIT); +const graph=()=>new Graph({nodes:500000,fuel:20000000,stack:50000},{threshold:100000}); +const list=(g,items)=>items.reduceRight((rest,item)=>g.fork(item,rest),g.leaf); + +test('indexed arithmetic and byte readers agree with independent numbers and malformed-byte controls',async()=>{ + const g=graph(),math=loadBundle(g,await bundle('tricu-math')),binary=loadBundle(g,await bundle('tricu-binary')); + const number=n=>natural(g,BigInt(n)),call=(library,name,...args)=>g.apply(library.get(name),args); + assert.equal(readNatural(g,call(math,'add',number(123456789),number(11))),123456800n); + for(const [divisor,q,r] of [[7,14,2],[0,0,100]]){ + const [quotient,remainder]=g.pair(call(math,'divMod',number(100),number(divisor))); + assert.equal(readNatural(g,quotient),BigInt(q));assert.equal(readNatural(g,remainder),BigInt(r)); + } + for(const [input,success,value,rest] of [[[65,66],true,65,[66]],[[256],false,15,[256]]]){ + const [flag,payload]=g.pair(call(binary,'readU8',list(g,input.map(number)))); + assert.equal(view(g,flag,'bool').value,success); + const [actual,remainder]=g.pair(payload);assert.equal(readNatural(g,actual),BigInt(value)); + assert.deepEqual(readList(g,remainder).map(v=>Number(readNatural(g,v))),rest); + } + for(const [input,expected] of [[[],true],[[195,169],true],[[192,175],false],[[237,160,128],false],[[226,130],false]]) + assert.equal(view(g,call(binary,'binaryUtf8?',list(g,input.map(number))),'bool').value,expected); +}); + +test('published predicate dispatch and list folds preserve first-match and early-stop demand boundaries',async()=>{ + const g=graph(),patterns=loadBundle(g,await bundle('tricu-patterns')),traversal=loadBundle(g,await bundle('tricu-list-traversal')); + const math=loadBundle(g,await bundle('tricu-math')),compiler=loadBundle(g,await bundle('tricu-compiler')); + const number=n=>natural(g,BigInt(n)),constant=value=>g.app(compiler.get('base.const'),value); + const predicate=g.app(math.get('gt'),number(5)); + const rules=list(g,[list(g,[predicate,g.app(math.get('add'),number(10))]),list(g,[constant(g.stem(g.leaf)),constant(number(99))])]); + for(const [input,expected] of [[3,13],[6,99]])assert.equal(readNatural(g,g.apply(patterns.get('match'),[number(input),rules])),BigInt(expected)); + assert.equal(g.toTernary(g.apply(patterns.get('match'),[number(0),g.leaf])),'0'); + const input=list(g,[number(1),number(2)]),stop=g.app(traversal.get('listStop'),number(7)); + const [state,rest]=g.pair(g.apply(traversal.get('listFoldUntil'),[input,number(0),constant(constant(stop))])); + assert.equal(readNatural(g,state),7n);assert.equal(g.force(rest),g.force(input),'Stop retains the current cell'); + const malformed=g.stem(g.leaf),q=g.fromTernary('212110021100'),divergent=g.app(q,q); + const [unchanged,suffix]=g.pair(g.apply(traversal.get('listFoldUntil'),[malformed,number(5),divergent])); + assert.equal(readNatural(g,unchanged),5n);assert.equal(g.force(suffix),malformed,'malformed tails halt without calling a divergent callback'); +}); diff --git a/ext/demo-site/test/release.py b/ext/demo-site/test/release.py @@ -1,6 +1,7 @@ #!/usr/bin/env python3 """Fast filesystem release/seed checks; no application compilation or evaluation.""" import argparse +import hashlib import importlib.util import json from pathlib import Path @@ -47,20 +48,28 @@ def main(): raise AssertionError('Expected integrity rejection') rejected(lambda: module.seed(pack, Path('relative-store'))) with tempfile.TemporaryDirectory(prefix='arboricx-seed-test-') as temp: - compiler_row = next(row for row in rows if row['uses'] == ['compiler']) + compiler_row = next(row for row in rows if 'compiler' in row['uses']) assert 1_048_576 < compiler_row['size'] <= 2 * 1_048_576 - # Role and size validation happens before any target is created. The - # compiler allowance must not leak into ordinary or mixed-role entries. + # The compiler ceiling applies even when that artifact is also a named + # release. Ordinary package roles independently admit exactly 16 MiB. limits_pack = Path(temp) / 'limits-pack' limits_pack.mkdir() - for changes in [dict(uses=['demo']), dict(uses=['compiler', 'demo']), - dict(uses=[]), dict(uses=['compiler', 'compiler']), - dict(uses='compiler'), dict(size=2 * 1_048_576 + 1)]: + for changes in [dict(uses=[]), dict(uses=['compiler', 'compiler']), + dict(uses='compiler'), dict(size=2_097_152 + 1), + dict(uses=['compiler', 'demo'], size=2_097_152 + 1), + dict(uses=['demo'], size=16_777_216 + 1)]: (limits_pack / 'dependencies.json').write_text(json.dumps( dict(schema=1, bundles=[dict(compiler_row, **changes)]))) target = Path(temp) / 'rejected-size-role' rejected(lambda: module.seed(limits_pack, target)) assert not target.exists() + data = b'x' * 16_777_216 + digest = hashlib.sha256(data).hexdigest() + obj = limits_pack / 'objects' / digest[:3] / (digest + '.arboricx') + obj.parent.mkdir(parents=True); obj.write_bytes(data) + (limits_pack / 'dependencies.json').write_text(json.dumps(dict(schema=1, + bundles=[dict(hash=digest, size=len(data), uses=['release:boundary@0.1.0'])]))) + assert module.seed(limits_pack, Path(temp) / 'boundary-store') == (1, 1) concurrent = Path(temp) / 'concurrent' children = [subprocess.Popen([str(args.deps / 'bin/arboricx-seed-site'), str(concurrent)], stdout=subprocess.PIPE, stderr=subprocess.PIPE) for _ in range(2)] diff --git a/ext/demo-site/test/showcase-library-probe.tri b/ext/demo-site/test/showcase-library-probe.tri @@ -0,0 +1,13 @@ +!import published_patterns as P +!import published_traversal as R +!import native_math as N +!export main + +-- Independent acceptance probe for published dispatch and early-stop folds. +dispatch value = P.match value [ + [(v : N.lt v 5) (v : N.add v 10)] + [(v : t t) (v : 99)]] +step 0 state = R.listStop state +step value state = R.listContinue (N.add state value) +main = [(dispatch 3) (dispatch 6) (P.match 0 []) + (R.listFoldUntil [1 2 0 9] 0 step)] diff --git a/ext/demo-site/test/showcase-preparation.test.mjs b/ext/demo-site/test/showcase-preparation.test.mjs @@ -13,9 +13,9 @@ site=Path(__import__('sys').argv[1]) spec=importlib.util.spec_from_file_location('showcase',site/'prepare-showcase.py') m=importlib.util.module_from_spec(spec);spec.loader.exec_module(m) index=json.loads((site/'packages.json').read_text()) -releases=[r for r in index['releases'] if r['name'] in ['tricu-graphs','parser-demo']] -demo=next(e for e in json.loads((site/'catalogue.json').read_text()) if e['id']=='parser') -goldens=json.loads((site/'parser-demo.json').read_text()) +releases=[r for r in index['releases'] if r['name'] in ['tricu-graphs','parser-demo','lexer-demo']] +demo=[e for e in json.loads((site/'catalogue.json').read_text()) if e['id'] in ['parser','lexer']] +goldens=[json.loads((site/(e['id']+'-demo.json')).read_text()) for e in demo] artifacts={r['hash']:site/'bundles'/(r['hash']+'.arboricx') for r in releases} with tempfile.TemporaryDirectory() as tmp: root=Path(tmp);(root/'bundles').mkdir() @@ -30,13 +30,22 @@ with tempfile.TemporaryDirectory() as tmp: assert (root/'packages.json').read_text()==before assert (root/'catalogue.json').read_text()=='[]\n' assert not list((root/'bundles').iterdir()) + # A conflict on the second demo must not discover the first or copy assets. (root/'packages.json').write_text(json.dumps(dict(schema=1,releases=[]))) + conflict=json.dumps([dict(demo[1],title='changed')]);(root/'catalogue.json').write_text(conflict) + try:m.stage(root,artifacts,releases,demo,goldens) + except ValueError:pass + else:raise AssertionError('conflicting second demo accepted') + assert (root/'catalogue.json').read_text()==conflict + assert not json.loads((root/'packages.json').read_text())['releases'] + assert not list((root/'bundles').iterdir()) + (root/'catalogue.json').write_text('[]') m.stage(root,artifacts,releases,demo,goldens) - assert json.loads((root/'catalogue.json').read_text())==[demo] - assert len(json.loads((root/'packages.json').read_text())['releases'])==2 + assert json.loads((root/'catalogue.json').read_text())==demo + assert len(json.loads((root/'packages.json').read_text())['releases'])==3 for pin,path in artifacts.items():assert (root/'bundles'/(pin+'.arboricx')).read_bytes()==path.read_bytes() m.stage(root,artifacts,releases,demo,goldens) # Exact retry is idempotent. - assert len(json.loads((root/'packages.json').read_text())['releases'])==2 + assert len(json.loads((root/'packages.json').read_text())['releases'])==3 print('accepted') `,site],{encoding:'utf8',timeout:15000,maxBuffer:100000}); assert.equal(output.trim(),'accepted');