Rebuilding the compiler

Compiler development uses an accepted stable compiler to build changed sources, then tests the resulting artifacts. At an agreed acceptance checkpoint, a fresh compiler builds itself again from the same frozen source. Independent audits verify both generations and their reproduced outputs.

For ordinary application authoring, use the included compiler and Build and run. Rebuilding uses either an explicit bootstrap kit containing the map-builder and standard IO adapter, or a completed generation from which those executables can be selected. The standalone application compiler has a different entrypoint.

Ordinary generations

Select these inputs explicitly:

Variable	Selection
TRICU	Nix-built Haskell executable.
PREVIOUS	Completed, audited generation supporting current native headers.
NEXT	New output directory in the working checkout or temporary storage.
ADAPTER	Verified standard six-argument IO adapter.
SOURCE	Source checkout to build.

An accepted-pair recovery transfer provides the completed second generation at recovery/seed and the adapter at recovery/tools/io-adapter.arboricx. Identify other inputs from their recorded provenance.

python3 test/bootstrap_generations.py "$PREVIOUS" "$NEXT" --tricu "$TRICU" \
  --source-root "$SOURCE" --backend stable --adapter "$ADAPTER" \
  --capabilities 8 --memory-mib 65536 --wall-seconds 7200
python3 test/bootstrap_store_check.py "$NEXT" --source-root "$SOURCE"

Add --expect-identical when rebuilding unchanged inputs. Omit it for source changes. The audit checks the complete selected dependency closure, including receipts and exported objects.

The runner uses the previous closure's compiled map-builder and precompiled adapter. tricu owns staging, compilation, graph policy, and receipts; Haskell executes IO, verifies transport, and packs bundles. Indexed compiler/adapter loading preserves sharing. Transport-object hashes and the compiler bundle identity used in receipts are recorded separately.

A full source_map generation produces:

• rebuilt-compiler.arboricx: map-builder for the next generation.
• build-stable.arboricx: stable application compiler, exporting sourceBuildGraph.
• build.arboricx: retained optimized application compiler under the same ABI.
• io-adapter.arboricx: inherited standard adapter.

Use --backend stable explicitly for new generations. The runner defaults to stable; historical records without a backend mean optimized. Generation selection uses sourceMapBuildStable or sourceMapBuild, while application selection uses the chosen application bundle. Both paths require the selected backend and entrypoint.

Execution-host selection

--execution-host zig --zig-library /nix/store/.../lib/libarboricx.so selects Zig execution through the owned ABI. --tricu selects the Haskell runtime for authenticated input transport and output packaging. The selected execution host performs compilation and publication; the default is Haskell. --backend independently selects the lowering backend. Missing libraries or exports fail preparation.

Provenance records the execution and transport runtimes separately. Historical records lacking execution_host describe Haskell execution. Zig records separate fingerprints for its shared library and Python launcher. The shared supervisor checks source consistency, independently audits the store, enforces publication and whole-job deadlines, and records completion status. Zig also applies an address-space ceiling, CPU ceiling, and an 8 MiB native stack.

Both execution routes grant broad file IO to trusted compiler jobs. Use the isolated workflow below to restrict filesystem access. Small-closure validation uses test/cross_host_generation.py, documented in the Zig runtime guide. Ordinary pair acceptance and retention toolchain roles use Haskell generations; the cross-host verifier compares Haskell and Zig generations.

Isolated cross-runtime fixed-point reproduction

The Linux workflow builds two Haskell generations, then uses the second compiler to build a Haskell control and a Zig-executed third generation:

seed --Haskell--> H1 --Haskell--> H2 --Haskell--> H3
                                   `----Zig----> Z3

nix develop -c python3 test/bootstrap_cross_host.py \
  --source-commit COMMITTED_SOURCE --directory "$SCRATCH/cross-runtime-new" \
  --tricu "$TRICU" --zig-library "$ZIG/lib/libarboricx.so"

Select exact Nix runtime paths and a new output directory. The source commit supplies the frozen map, sources, kit and inner generation runner. Each job uses a fresh store in a filesystem sandbox containing its source snapshot, selected compiler/adapter kit, Nix dependencies and writable work directory. Haskell performs common transport and packing; the selected execution host performs compilation and publication. Jobs run sequentially with fixed limits of 32 GiB and one hour per job; Haskell uses eight capabilities. See the isolated probe for namespace requirements and bounded validation before a full run.

experiment.json records source/archive hashes, mapped inputs, runtime/Python identities, tooling and comparison boundaries. Logs, invocations and runtime provenance identify each execution separately. The independent verifier checks H1/H2 and H2/H3 acceptance, verifies that Z3 used H2's exact compiler, and compares the actual H2/H3/Z3 object inventories, module checkpoints/receipts and all three packaged compiler artifacts. accepted-cross-host.json records the verified reproduction result. --verify-only rechecks existing artifacts and provenance.

Post-generation validation runs the H2 program smoke and checks its complete results on Zig:

nix develop -c python3 test/bootstrap_smoke.py "$TRICU" \
  --generation "$EXPERIMENT/H2/generation" \
  --acceptance "$EXPERIMENT/accepted-pair.json" \
  --source-root "$EXPERIMENT/source" --directory "$EXPERIMENT/haskell-smoke"
nix develop -c python3 test/bootstrap_cross_host_smoke.py "$EXPERIMENT" \
  --library "$ZIG/lib/libarboricx.so" --directory "$EXPERIMENT/zig-program-smoke"
nix develop -c python3 test/bootstrap_cross_host_controls.py "$EXPERIMENT" \
  --directory "$SCRATCH/cross-host-controls-new"

The program smoke covers native imports, recursive lists, and invoice totals and decimal text against independent complete trees. Haskell executes the program's compilation using the bit-identical accepted compiler; both hosts execute the resulting program. Negative controls mutate independent copies and require the verifier to reject false host/runtime/freshness/compiler claims, corrupt objects, changed packaged artifacts and additional valid objects. Restored copies must pass again.

Finish rolling retention with H2 as the current Haskell toolchain and an independently useful rollback. Protect H3/Z3, frozen inputs, cross-host reports, and a registered Nix root for the Zig runtime as additional evidence. Common preparation, transport and packaging use Haskell throughout this workflow.

Explicit bootstrap kit

The repository includes the accepted kit at bootstrap/rebuild/. It works from a source archive without Git history or a previous private store. Set KIT="$PWD/bootstrap/rebuild" when rebuilding that archive.

A kit is a caller-selected directory containing bootstrap-kit.json, rebuilt-compiler.arboricx, and io-adapter.arboricx. Select it without a previous store:

python3 test/bootstrap_generations.py "$FIRST" --bootstrap-kit "$KIT" \
  --tricu "$TRICU" --source-root "$FROZEN_SOURCE" --backend stable --fresh \
  --capabilities 8 --memory-mib 65536 --wall-seconds 7200

The JSON manifest has schema tricu.bootstrap-kit.v1, backend stable, and an artifacts object with exactly two roles:

Role	file	export
compiler	rebuilt-compiler.arboricx	sourceMapBuildStable
adapter	io-adapter.arboricx	main

Each role also records its lowercase whole-file sha256. Select the manifest through a trusted release or verified artifact preparation. The runner verifies both files before creating output, captures their bytes, imports them indexed, and records the manifest digest alongside the actual executable identities. The kit supplies the complete compiler selection, independently of previous stores. A --validate-only run checks source/map staging inputs without loading executables.

Use generation one's completed output as PREVIOUS for generation two with --expect-identical. Kit selection cannot be combined with previous-generation, adapter, or transition-worker overrides. Prepare an explicit kit with the verified transition worker when the batch needs one. The fresh-pair acceptance, independent audits, and publication requirements below still apply.

Development batches and acceptance checkpoints

Agree on a coherent change and focused-check budget before implementation. Compile changed modules and affected consumers with the accepted stable compiler. Check complete results, diagnostics, demand, effects, integrity, and relevant workloads against independent expectations. Testing distinguishes these candidate checks from reference-artifact tests and acceptance.

Several reviewable candidate commits may precede a generation checkpoint. Record the candidate identities, checks, pending freshness, and acceptance scope in STATUS.md. Keep accepted fixtures and the rollback toolchain unchanged until publication. Use focused checks between drafts and run full generations at the agreed checkpoint.

Verification failures block acceptance and further feature work. Diagnose and repair with bounded checks. A failed generation gate requires a passing fresh pair for acceptance, including complete independent audits. Cached equality and refreshed fixtures serve only as supporting evidence.

Mandatory compiler hygiene gate

For a compiler-affecting batch, freeze source at an exact Git commit. Syntax, semantics, lowering, dependencies, staging, and compiler transport all affect this boundary. Select FROZEN_SOURCE, new FIRST and SECOND directories, and a new ACCEPTANCE report path:

python3 test/bootstrap_generations.py "$PREVIOUS" "$FIRST" --tricu "$TRICU" \
  --source-root "$FROZEN_SOURCE" --backend stable --fresh --adapter "$ADAPTER" \
  --capabilities 8 --memory-mib 65536 --wall-seconds 7200
python3 test/bootstrap_generations.py "$FIRST" "$SECOND" --tricu "$TRICU" \
  --source-root "$FROZEN_SOURCE" --backend stable --fresh --expect-identical \
  --capabilities 8 --memory-mib 65536 --wall-seconds 7200
python3 test/bootstrap_acceptance.py "$FIRST" "$SECOND" --tricu "$TRICU" \
  --output "$ACCEPTANCE"

Both output directories must be nonexistent. Generation two must actually use generation one's rebuilt compiler on the same frozen inputs. A transition worker may be selected for generation one only. Acceptance requires independent closure/receipt audits, identical executable manifests, and exact reproduction of all three compiler/application bundles.

Run generation-two correctness smokes separately:

python3 test/bootstrap_smoke.py "$TRICU" --generation "$SECOND" \
  --acceptance "$ACCEPTANCE" --source-root "$FROZEN_SOURCE" \
  --directory "$SMOKE_DIRECTORY"

The smoke probe authenticates its compiler/runtime and mapped library inputs, builds a fresh workspace, audits sources/dependencies/exports, and checks native imports, patterns, top-level/local recursion, and a small invoice program. Its whole-job limit is five minutes with an 8 GiB sampled child guard. Self-reproduction and these smokes complement the batch's independent semantic tests.

Verified compiler checkpoint; publication deferred

An explicitly agreed checkpoint may promote the audited fresh pair after the smokes and rolling retention while deferring fixture publication, the broad downstream sweep, and recovery packaging. Record verified compiler checkpoint, publication deferred, with source commit, tool identities, coverage, and remaining obligations. Retain the previous accepted compiler for rollback. Promotion covers the development compiler; support-cohort acceptance and fixture publication remain pending.

Publish the accepted batch

When replacing the shipped bootstrap/build-stable.arboricx, update its trusted whole-file SHA256 in src/PortableCompiler.hs (packagedCompilerSha256) alongside bootstrap/stable-compiler.json. The packaged frontend rejects mismatched bytes; its receipt identity is this bundle digest. Indexed transport objects have their own separately recorded hashes.

Publish genuinely rebuilt compiler and support fixtures using the procedures in Testing. Capture matching source snapshots, then check freshness and the fast gate:

python3 test/fixture_sources.py capture --source-root "$FROZEN_SOURCE"
python3 test/fixture_sources.py current --source-root "$FROZEN_SOURCE"
nix flake check

Reference snapshots authenticate the bytes actually used to compile an artifact. Publish freshly compiled artifacts with fingerprints of their actual inputs. Support-only .tri changes require candidate/publication checks; compiler generations are required for compiler-affecting changes.

Full compiler publication also includes the browser/site assets. Commit the accepted compiler/support fixtures first (the site authenticates their full Git origin), then complete the site compiler-promotion refresh and commit its matching assets and validation changes. Update the site’s separately pinned compiler explicitly. Full publication remains incomplete while the site refresh is deferred. Finish rolling retention after this refresh; retain matching previous site/dependency outputs for rollback. Public deployment requires separate authorization after local preparation and validation.

Inputs, restart, and resource controls

Full jobs use eight GHC capabilities, at most 64 GiB, and a two-hour whole-job ceiling. Select --capabilities 8 --memory-mib 65536 --wall-seconds 7200 explicitly; CLI defaults are smaller. Use lower bounds for focused probes. Measure the effect of capability counts on the selected workload.

On an interactive workstation, run heavy jobs sequentially with low priority. If selecting CPU affinity, choose available CPUs with a caller-provided CPU_SET. Configure Nix daemon builders separately with Nix’s job/core settings or builder limits; caller affinity applies to the caller’s processes.

Option	Meaning
--map, --source-root, --root	Explicit map, source location, and closure root.
--validate-only	Validate staging inputs without executing a build.
--memory-mib	Sampled direct-child RSS guard and host heap ceiling; default 32768, maximum 65536 MiB.
--capabilities	GHC capabilities; default one.
--initial-seconds, --extended-seconds	No-publication boundaries.
--wall-seconds	Whole-job budget including preparation, audits, and packing; default 3600, maximum 7200.

Native transport commands also have a 600-second timeout. RSS sampling covers the direct child; descendants require separate aggregate limits. Sampled peaks can miss short-lived allocations. Keep the enclosing process alive beyond the configured guards.

A native map row preserves source bytes and declared import/export policy. Its ordered dependencies must match local imports, re-exports, and build-only edges, deduplicated in first-occurrence order. Legacy unit and reexport rows translate checked headers. Compiler dependency names are map aliases; supply them through a staged workspace or immutable manifests.

Sources must remain unchanged throughout a generation. The runner checks them again before accepting outputs. Snapshot consistency relies on trusted, stable filesystem contents throughout the job. inputs.json, source-map.v1, invocation.json, logs, and status.json record input identities, limits, comparisons, and completion.

Rerunning a diagnostic output directory reuses verified checkpoints. Changed source/compiler identities invalidate keys, and an output lock prevents concurrent writers. Interrupted builds remain incomplete. Resumption helps diagnosis and recovery; the acceptance pair requires fresh stores. The map-builder checkpoints completed modules as whole units.

Transitioning older inputs

An older closure may lack the selected entrypoint, current native headers, or verified native tree transport. Prepare an explicit bridge using a retained portable application compiler and standard adapter:

python3 test/prepare_generation_worker.py "$TRICU" "$PREVIOUS" "$WORKER" \
  --compiler-bundle "$APPLICATION_COMPILER" --backend stable
python3 test/bootstrap_generations.py "$PREVIOUS" "$NEXT" --tricu "$TRICU" \
  --source-root "$SOURCE" --backend stable --adapter "$ADAPTER" \
  --worker-bundle "$WORKER/worker.arboricx" \
  --capabilities 8 --memory-mib 65536 --wall-seconds 7200

Worker preparation selects only the requested backend and its reachable local definitions. It audits every transported support export. Inspect --plan-only before preparing a retained flat compiler; its source compiler export can exceed the normal 64 MiB flat limit. Select an explicit --support-flat-bytes BYTES for that verified support transport. Ordinary flat imports retain their default 64 MiB limit. Worker compilation has a 900-second guard within the selected whole-job limit.

For a pre-consolidation header parser, explicitly add --legacy-compiler-header to worker preparation. It changes only the bridge wrapper’s header; native language semantics and failed-compilation behavior remain unchanged. Subsequent generations use the rebuilt closure directly. Both routes require the standard six-argument adapter. Test wrappers also compile through retained portable tools.

Rolling development retention

Retain the latest accepted second-generation store and the preceding accepted second-generation store: two compiler revisions for development and rollback. Keep complete CAS/cache/staging inputs, frozen source/maps, receipts, bundles, adapters, acceptance reports, and the exact Haskell runtime closure protected from Nix GC. Also retain current support preparation and committed fixtures.

Before pruning an older store:

1. Audit both retained closures and source snapshots.
2. Hide older stores. Exercise imports, compilation, execution, diagnostics, integrity controls, and a fresh scoped map-builder job with each retained compiler and adapter.
3. Authenticate current support preparation and reproduce its complete manifests without the old stores, within the selected audit budget.
4. Preserve compact sources, bundles, receipts/manifests, and reports. Record the exact removed object directories and mark their parents incomplete. Unlink obsolete paths, preserving the bytes of shared hardlinked objects.
5. Re-audit retained stores and smoke-test both compilers after deletion.

Check for sibling-worktree users before pruning. Removing generation one means its original full-pair audit cannot be rerun in place; preserve that distinction in the retained evidence. Off-site recovery packages remain outside routine scratch cleanup.

Checkpoint cleanup command

Checkpoint completion requires pair acceptance, smokes, and rolling retention. Run test/bootstrap_retention.py after acceptance and smokes, before declaring completion or starting the next feature batch. This opt-in Linux/Haskell workflow requires Python, Nix and bwrap. It validates retained toolchains and prunes selected scratch objects; full compiler generation and Nix GC are separate operations. An existing verified checkpoint can satisfy its outstanding retention obligation without building a replacement pair.

Create an explicit local retention registry, for example $SCRATCH/retained-toolchains.json. Replace the example paths and runtime hashes with recorded provenance as the basis for compiler selection. current and rollback are required unique roles. Additional uniquely named toolchains or protected scratch directories are allowed for active consumers and release evidence. Prefer an independently useful prior compiler revision for executable rollback. Two builds with identical bundles retain the same executable revision.

{
  "schema": "tricu.bootstrap-retention.v1",
  "protected": ["current-cohort", "previous-cohort"],
  "sibling_users_checked": true,
  "uncoordinated_jobs_stopped": true,
  "toolchains": [
    {
      "role": "current",
      "generation": "current-cohort/generation2",
      "source": "current-cohort/source",
      "acceptance": "current-cohort/accepted-pair.json",
      "runtime": "/nix/store/CURRENT-RUNTIME/bin/tricu"
    },
    {
      "role": "rollback",
      "generation": "previous-cohort/generation2",
      "source": "previous-cohort/source",
      "acceptance": "previous-cohort/accepted-pair.json",
      "runtime": "/nix/store/PREVIOUS-RUNTIME/bin/tricu"
    }
  ],
  "support": {
    "toolchain": "current",
    "preparation": "current-cohort/support",
    "relocate_patterns": true
  }
}

All scratch paths are normalized, root-relative, and protected by explicit real directories. The current command reruns full-pair acceptance, so both retained first generations must also be protected and present throughout retention. Acceptance must select the actual retained second-generation path; for a relocated pair, create relocated evidence with bootstrap_acceptance.py and its explicit source override. Preserve the original acceptance records. The selected standard adapters remain inside the protected generations.

The support preparation must contain the complete selected candidate cohort in one directory. The command reproduces current support bundles, metadata, receipts and full native manifests through the existing publisher, then checks source freshness. Published fixtures remain unchanged. relocate_patterns selects the current V3 publication procedure. Protect any other preparation or store used by active work before planning. The operator must check external users and stop uncoordinated jobs before setting the two confirmation fields.

SCRATCH="$PWD/.bootstrap-run"
CHECKPOINT="$SCRATCH/retention-checkpoint-UNIQUE"
python3 test/bootstrap_retention.py plan --root "$SCRATCH" \
  --config "$SCRATCH/retained-toolchains.json" --directory "$CHECKPOINT"

# Review plan.json: protected inputs, exact objects/ paths, compact evidence,
# and allocated-byte reclaim estimate. Use the SHA256 printed by plan.
python3 test/bootstrap_retention.py apply --root "$SCRATCH" \
  --directory "$CHECKPOINT" --approve-plan REVIEWED_PLAN_SHA256

# Mandatory completion gate: nonzero while pending or evidence/inputs differ.
python3 test/bootstrap_retention.py check --root "$SCRATCH" \
  --directory "$CHECKPOINT"

Planning registers indirect Nix GC roots for each exact accepted Haskell runtime. It inventories allocated blocks by inode, excludes hardlinks still referenced by retained or external files from reclaim estimates, and preserves compact header-bearing CAS documents with whole-file hashes before any deletion. Only selected objects/ directories inside this .bootstrap-run are candidates. Sources, portable bundles, original reports and archives outside that scope are preserved. Each removed CAS gets a PRUNED-CAS.json marker; its old completion records serve as historical evidence, and its store is marked incomplete.

Application holds an exclusive scratch-root lock. Generation runners take shared locks on previous/output scratch roots, so they refuse concurrent cleanup. Existing generation output locks are checked too. Stop other tools and external users separately: lock coordination covers the generation runners and retention command. Use trusted local filesystems and keep adversarial writers or mounts outside the workflow.

Before deletion, bwrap hides the entire scratch root, exposing only registered protected directories and the writable checkpoint evidence directory. Each selected compiler passes independent pair audits, CLI compilation/execution, diagnostics and integrity controls, correctness smokes, and a fresh scoped native_math map-builder/adapter job with complete-manifest comparison. Support reproduction must pass in that isolation too. The same checks run after deletion; only then can the checkpoint's plan.json become complete: true and the completion gate succeed. Verification logs, reports, registry and harness snapshots remain beside the plan. Keep those evidence directories and their runtime roots.

Deletion is journaled and resumable. A failed precheck deletes nothing. A failed postcheck leaves retention incomplete even if unlinking finished. For resumption, review the updated plan and approve its current SHA256; changed protected inputs, candidates, compact evidence or deletion harness block resumption. The completion checker can read archived harness identities after tooling evolves, but an in-flight delete must use its reviewed harness. Report an incomplete checkpoint with its blockers and outstanding bytes. Perform rotations at each checkpoint, while sufficient disk space remains.

Safety controls run in the fast Nix check without compiling tricu programs:

nix build .#checks.x86_64-linux.bootstrapRetention --no-link

Recovery archives

Select a completed generation, its matching frozen commit, standard adapter, Haskell executable, and a new transfer directory:

python3 test/bootstrap_archive.py \
  --generation "$PREVIOUS" --seed-commit "$MATCHING_COMMIT" \
  --adapter "$ADAPTER" --tricu "$TRICU" --output "$TRANSFER_DIRECTORY"

For a verified pair, also supply --first-generation "$FIRST" and --acceptance "$PAIR_ACCEPTANCE", with --generation selecting generation two. Current fixture fingerprints must match current mapped source. The selected seed's source must match its recorded inputs exactly.

The transfer includes full generation stores and provenance, matching source, current tracked working files, reachable Git history/refs, a tracked-change patch, fixtures, tools, and the Nix runtime closure. Accepted-pair transfers preserve actual reproduced tools; seed-only transfers pack current fixture entrypoints. Untracked scratch files are excluded. The historical Zig build-output symlink is omitted from source snapshots and retained only in Git history.

Creation verifies packed tools, extracts independently, checks all checksums, repeats relocated audits, verifies Git history, reimports the runtime closure, and probes native imports and stable recursion in fresh stores. Allow space for uncompressed stores and temporary validation. An interrupted transfer is incomplete until its final checksum and validation sidecars exist.

RECOVERY.json records inputs and limits; RESTORE-VALIDATION.json records actual checks. Follow the transfer's own README when restoring. Verify TRANSFER-SHA256SUMS before extraction and internal SHA256SUMS afterward. Existing transfers remain immutable when documentation changes.

Archive creation packages and validates existing accepted evidence. Runtime reimport is checked on the same host; empty-Nix-store and cross-platform recovery require separate validation. The retained Nix closure supports running the host; an offline Haskell rebuild requires its build dependencies as well. Archived SCP is retained solely as historical material.

Copy the whole transfer directory, including checksum and validation sidecars, to independent off-machine storage, and verify restoration before relying on it. Maintain an independently stored backup and retain the original recovery transfers.

Focused validation

python3 test/bootstrap_generation_cli.py "$PREVIOUS" --backend stable --adapter "$ADAPTER"
python3 test/verified_tree_loading.py "$TRICU" "$PREVIOUS" \
  "$DIAGNOSTIC_GENERATION" --compiler-bundle "$APPLICATION_COMPILER" \
  --directory "$PROBE_DIRECTORY" --publish --imports \
  --memory-mib 32768 --seconds 600

Use new writable diagnostic directories. The CLI probe checks cold/warm/change, interruption, corruption, backend selection, and receipt options. An older closure can use an explicit --worker-bundle. The transport probe compiles an adapter through the selected portable toolchain and checks unit exports and loading. Its checkpoints support diagnosis; ordinary generation completion requires the generation runner and acceptance workflow above. See Storage for the transport contract.