tricu

An interpreted language for exploring Tree Calculus
Log | Files | Refs | README | LICENSE

commit f42db2d18a96b8cb19d7bbf8bcac06a2f0726351
parent e68c7cfc3b7a3e3ecf8cfefbb1ec883c21abd663
Author: James Eversole <james@eversole.co>
Date:   Fri, 16 Oct 2026 17:52:01 -0500

Reduce Zig evaluator allocation and dispatch overhead with bounded parity checks

Diffstat:
Mext/zig/README.md | 66++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++--
Mext/zig/include/arboricx.h | 6++++++
Mext/zig/src/arena.zig | 107+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++------------------
Mext/zig/src/c_abi.zig | 4++++
Mext/zig/src/indexed.zig | 2+-
Mext/zig/src/io_driver.zig | 9+++++----
Mext/zig/src/reduce.zig | 212+++++++++++++++++++++++++++++++++++++++++++++++++++++--------------------------
Mext/zig/src/reduce_test.zig | 341++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++-
Mext/zig/src/source_text.zig | 4++--
Mext/zig/tests/handles_test.c | 3+++
Mtest/bootstrap_generations.py | 9+++++++++
Mtest/cross_host_generation.py | 11++++++++++-
Mtest/zig_generation_executor.py | 8++++++++
13 files changed, 674 insertions(+), 108 deletions(-)

diff --git a/ext/zig/README.md b/ext/zig/README.md @@ -83,6 +83,61 @@ before execution. Runtime Debug/ReleaseSafe tests cover source/indexed IO permission denial and malformed inputs. The execution child records string diagnostics or a bounded numeric prefix for structured diagnostics. +## Evaluator allocation headroom + +The default collector runs after `max(1048576, last_live_nodes)` allocations +since the last collection. It recomputes this threshold after each collection, +so a transient live-heap peak does not permanently delay later collections. +Construction still never collects automatically. These thresholds count nodes, +not bytes, and are not RSS limits. + +C ABI clients can choose the memory/speed tradeoff per context: + +```c +/* More allocation headroom for a memory-rich compiler job. Check the status. */ +arb_status_t status = arb_set_gc_policy(ctx, 4194304, 4); +``` + +The two positive `uint32_t` arguments are the minimum allocation interval and +live-node multiplier. The threshold is their maximum after multiplying the +most recently measured live-node count (with saturating arithmetic). Changing +policy does not allocate, collect, or reset the allocation counter. The next +reducer safepoint checks the new threshold. Zero arguments return +`ARB_INVALID_DATA` without changing the policy. `arb_collect` still requests an +explicit collection regardless of the threshold. Older ABI-v2 libraries may +lack this additive entrypoint; callers requiring it must select a library that +exports it, not silently fall back. + +The generation runner accepts the same explicit policy and records it in input +provenance, status, invocation and the execution report. For a bounded native +math compilation, using the runtime paths from the small compiler probe: + +```sh +nix develop -c python3 test/bootstrap_generations.py /tmp/zig-math-headroom-new \ + --bootstrap-kit bootstrap/rebuild --tricu "$HASKELL/bin/tricu" \ + --execution-host zig --zig-library "$ZIG/lib/libarboricx.so" \ + --root native_math --fresh --memory-mib 4096 --wall-seconds 180 \ + --zig-gc-policy 4194304 4 +``` + +Use a fresh directory for each sample. Omit `--zig-gc-policy` to measure the +library default; it is also accepted by `test/cross_host_generation.py` for +cold/warm parity and compatibility checks. The existing isolated 1 GiB probe +continues to use the default. Larger headroom may need several GiB, even on a +single module; increasing it further need not improve execution time. Keep +external resource guards and measure both time and peak RSS on the actual job. +The collector retains backing capacity, including its mark/work buffers, and +the reducer retains continuation-buffer capacity. Collection trims dead node +array tails without moving live IDs, but does not promise to return memory to +the OS. + +For comparisons, alternate original/candidate order on one allowed CPU after +builds finish. Compare `execution.json` execution times and `status.json` child +peak RSS, not just supervisor wall time. Require independent audits and identical +complete `cas/objects` inventories. A native-math measurement is not a full +compiler-generation measurement; use the cross-runtime reproduction workflow +for compiler fixed-point acceptance. + ## Indexed tree transport The IO actions `readIndexedTree` (50), `hashIndexedTree` (51), and @@ -187,8 +242,15 @@ arboricx_free(ctx); ## Reduction and collection contract The reducer is a heap-framed, in-place, call-by-need WHNF machine. It evaluates -terms on demand. Suspended function, discriminator, argument and update demands -use heap frames. +terms on demand. Constructor demands dispatch directly; only suspended function, +discriminator, argument and update demands use heap frames. Already-normal +observations avoid borrowing continuation storage unless collection is due. +Distribution also fuses immediately demanded constructor/constant applications, +without allocating their private intermediate thunks. These are kernel-rule +reductions, not source-name intrinsics: they do not demand discarded arguments, +and charge the same application-entry fuel as the unfused rules. Low-fuel cases +fall back to the unfused path. Tests compare single-invocation fuel boundaries +and resumed results against a literal unfused evaluator. The arena uses non-moving mark/sweep and reuses dead slots. Collection occurs at reducer loop safepoints, with the current term and continuation redexes rooted. diff --git a/ext/zig/include/arboricx.h b/ext/zig/include/arboricx.h @@ -31,6 +31,12 @@ uint32_t arb_abi_version(void); arb_status_t arb_retain(arb_ctx_t* ctx, arb_handle_t handle, arb_handle_t* out); arb_status_t arb_release(arb_ctx_t* ctx, arb_handle_t handle); arb_status_t arb_collect(arb_ctx_t* ctx); +/* Set allocation headroom: max(minimum, last_live_nodes * multiplier). + * Both arguments must be positive; invalid input leaves the policy unchanged. + * No collection/allocation occurs here. The next reducer safepoint checks the + * new threshold against allocations since the last collection. Default: 1048576, + * 1. This is NOT a memory limit; construction does not automatically collect. */ +arb_status_t arb_set_gc_policy(arb_ctx_t* ctx, uint32_t minimum, uint32_t multiplier); /* Context lifecycle */ arb_ctx_t* arboricx_init(void); diff --git a/ext/zig/src/arena.zig b/ext/zig/src/arena.zig @@ -1,6 +1,11 @@ const std = @import("std"); const tree = @import("tree.zig"); +pub const ReductionFrame = struct { + kind: enum { function, discriminator, argument, update }, + redex: u32, +}; + pub const Arena = struct { allocator: std.mem.Allocator, nodes: std.ArrayList(tree.Node), @@ -9,8 +14,17 @@ pub const Arena = struct { roots: std.ArrayList(u32) = .empty, free: std.ArrayList(u32) = .empty, allocations: usize = 0, - gc_interval: usize = 65536, + // Allocation headroom, not a memory limit. Tests can select a small policy + // to force collections at suspension/allocation boundaries. + gc_min_interval: usize = 1 << 20, + gc_live_multiplier: usize = 1, + gc_interval: usize = 1 << 20, collections: usize = 0, + last_live: usize = 0, + marks: std.ArrayList(bool) = .empty, + grey: std.ArrayList(u32) = .empty, + reduction_frames: std.ArrayList(ReductionFrame) = .empty, + reduction_roots: std.ArrayList(u32) = .empty, pub fn init(allocator: std.mem.Allocator) Arena { return .{ @@ -23,21 +37,40 @@ pub const Arena = struct { self.nodes.deinit(self.allocator); self.roots.deinit(self.allocator); self.free.deinit(self.allocator); + self.marks.deinit(self.allocator); + self.grey.deinit(self.allocator); + self.reduction_frames.deinit(self.allocator); + self.reduction_roots.deinit(self.allocator); } - pub fn alloc(self: *Arena, node: tree.Node) !u32 { - if (self.free.pop()) |idx| { - self.nodes.items[idx] = node; + inline fn allocSlot(self: *Arena) !u32 { + if (self.free.items.len != 0) { + const idx = self.free.items[self.free.items.len - 1]; + self.free.items.len -= 1; self.allocations += 1; return idx; } if (self.nodes.items.len >= std.math.maxInt(u32)) return error.OutOfMemory; const idx: u32 = @intCast(self.nodes.items.len); - try self.nodes.append(self.allocator, node); + _ = try self.nodes.addOne(self.allocator); self.allocations += 1; return idx; } + pub inline fn alloc(self: *Arena, node: tree.Node) !u32 { + const idx = try self.allocSlot(); + self.nodes.items[idx] = node; + return idx; + } + + // Reduction allocates applications in its hottest path. Scalar operands + // avoid constructing and then copying a temporary padded tagged union. + pub inline fn allocApp(self: *Arena, func: u32, arg: u32) !u32 { + const idx = try self.allocSlot(); + self.nodes.items[idx] = .{ .app = .{ .func = func, .arg = arg } }; + return idx; + } + pub fn get(self: *Arena, idx: u32) *tree.Node { return &self.nodes.items[idx]; } @@ -50,42 +83,68 @@ pub const Arena = struct { return self.gc_enabled and self.allocations >= self.gc_interval; } + pub fn setGcPolicy(self: *Arena, minimum: usize, multiplier: usize) error{InvalidData}!void { + if (minimum == 0 or multiplier == 0) return error.InvalidData; + self.gc_min_interval = minimum; + self.gc_live_multiplier = multiplier; + self.gc_interval = @max(minimum, self.last_live *| multiplier); + } + + fn mark(self: *Arena, idx: u32) !void { + if (self.marks.items[idx]) return; + // Mark on enqueue so shared children occupy the work stack only once. + try self.grey.append(self.allocator, idx); + self.marks.items[idx] = true; + } + /// Non-moving mark/sweep. Only reduction safepoints collect automatically; /// construction never collects, so newly allocated children can be linked /// without temporary root registrations. Failure before sweep changes no IDs. pub fn collect(self: *Arena, temporary_roots: []const u32) !void { - const marks = try self.allocator.alloc(bool, self.nodes.items.len); - defer self.allocator.free(marks); - @memset(marks, false); - var grey: std.ArrayList(u32) = .empty; - defer grey.deinit(self.allocator); - try grey.appendSlice(self.allocator, self.roots.items); - try grey.appendSlice(self.allocator, temporary_roots); + try self.marks.resize(self.allocator, self.nodes.items.len); + @memset(self.marks.items, false); + self.grey.clearRetainingCapacity(); + for (self.roots.items) |idx| try self.mark(idx); + for (temporary_roots) |idx| try self.mark(idx); var live: usize = 0; - while (grey.pop()) |idx| { - if (marks[idx]) continue; - marks[idx] = true; + var end: usize = 0; + while (self.grey.pop()) |idx| { live += 1; + end = @max(end, @as(usize, idx) + 1); switch (self.nodes.items[idx]) { .leaf => {}, - .stem => |s| try grey.append(self.allocator, s.child), - .fork => |f| try grey.appendSlice(self.allocator, &.{ f.left, f.right }), - .app => |a| try grey.appendSlice(self.allocator, &.{ a.func, a.arg }), + .stem => |s| try self.mark(s.child), + .fork => |f| { + try self.mark(f.left); + try self.mark(f.right); + }, + .app => |a| { + try self.mark(a.func); + try self.mark(a.arg); + }, } } - try self.free.ensureTotalCapacity(self.allocator, self.nodes.items.len - live); - self.free.clearRetainingCapacity(); - var i = self.nodes.items.len; + try self.free.ensureTotalCapacity(self.allocator, end - live); + // Free-list entries outside its logical length are never read. Rebuild + // without poisoning the previous list only to overwrite it immediately. + self.free.items.len = 0; + // Dead trailing slots need neither sweeping nor free-list entries. + // IDs of live nodes never change; backing capacity remains available. + // Slice truncation keeps bounds checks without touching dead backing + // storage. alloc/allocApp initialize a recycled slot before publishing it. + self.nodes.items = self.nodes.items[0..end]; + var i = end; while (i > 0) { i -= 1; - if (!marks[i]) { - self.nodes.items[i] = .leaf; + if (!self.marks.items[i]) { self.free.appendAssumeCapacity(@intCast(i)); } } + self.last_live = live; self.allocations = 0; self.collections += 1; // Amortize scanning a large live graph; this is not a hard RSS bound. - self.gc_interval = @max(self.gc_interval, live); + // Recompute rather than retaining a transient live-heap peak forever. + self.gc_interval = @max(self.gc_min_interval, live *| self.gc_live_multiplier); } }; diff --git a/ext/zig/src/c_abi.zig b/ext/zig/src/c_abi.zig @@ -90,6 +90,10 @@ export fn arb_retain(ctx: *ArbCtx, handle: Handle, out: *Handle) Status { const node = ctx.handles.get(handle) catch |err| return status(err); return ctx.publish(node, out); } +export fn arb_set_gc_policy(ctx: *ArbCtx, minimum: u32, multiplier: u32) Status { + ctx.arena.setGcPolicy(minimum, multiplier) catch |err| return status(err); + return .ok; +} export fn arb_collect(ctx: *ArbCtx) Status { ctx.prepare() catch |err| return status(err); ctx.arena.collect(&.{}) catch |err| return status(err); diff --git a/ext/zig/src/indexed.zig b/ext/zig/src/indexed.zig @@ -317,7 +317,7 @@ test "physical sharing is only an acceleration hint, and lazy children are seria const a = try encode(&arena, shared); defer arena.allocator.free(a); arena.gc_enabled = true; - arena.gc_interval = 1; + try arena.setGcPolicy(1, 1); const b = try encode(&arena, copied); defer arena.allocator.free(b); try std.testing.expectEqualSlices(u8, a, b); diff --git a/ext/zig/src/io_driver.zig b/ext/zig/src/io_driver.zig @@ -1348,11 +1348,12 @@ test "scheduler safepoint roots include blocked tasks, frames, environment, stat try blocked.frames.append(arena.allocator, .{ .local = saved_env }); try blocked.frames.append(arena.allocator, .{ .bind = continuation }); for (0..1000) |_| _ = try arena.alloc(.leaf); + const before_collection = arena.len(); arena.gc_enabled = true; - arena.gc_interval = 1; + try arena.setGcPolicy(1, 1); try scheduler.stepTask(task); try testing.expect(arena.collections > 0); - try testing.expect(arena.free.items.len >= 1000); + try testing.expect(before_collection - arena.len() + arena.free.items.len >= 1000); try testing.expectEqual(@as(usize, 1), arena.roots.items.len); // Observations below are outside a scheduler turn; explicitly hold its // roots by stopping automatic collection, not by accidentally relying on @@ -1379,7 +1380,7 @@ test "IO sentinel forces delayed outer and version/action pair" { const k_sentinel = try arena.alloc(.{ .fork = .{ .left = leaf, .right = sentinel } }); const program = try arena.alloc(.{ .app = .{ .func = k_sentinel, .arg = leaf } }); arena.gc_enabled = true; - arena.gc_interval = 1; + try arena.setGcPolicy(1, 1); const result = try runIO(arena.allocator, &arena, program, .{}); arena.gc_enabled = false; // result is held locally, outside a registered scope try testing.expectEqual(@as(u64, 42), (try codecs.toNumber(&arena, result)).?); @@ -1391,7 +1392,7 @@ fn testAction(arena: *Arena, tag: u64, payload: u32, perms: IOPerms) !codecs.Unw const rest = try arena.alloc(.{ .fork = .{ .left = try codecs.ofNumber(arena, 1), .right = action } }); const program = try arena.alloc(.{ .fork = .{ .left = try codecs.ofString(arena, "tricuIO"), .right = rest } }); arena.gc_enabled = true; - arena.gc_interval = 1; + try arena.setGcPolicy(1, 1); const result = try runIO(arena.allocator, arena, program, perms); arena.gc_enabled = false; // observations are local, outside the scheduler return (try codecs.unwrapResult(arena, result)).?; diff --git a/ext/zig/src/reduce.zig b/ext/zig/src/reduce.zig @@ -1,23 +1,32 @@ -const std = @import("std"); const Arena = @import("arena.zig").Arena; pub const ReduceError = error{ FuelExhausted, InvalidApply, OutOfMemory }; -const Frame = struct { - kind: enum { function, discriminator, argument, update }, - redex: u32, -}; +const Frame = @import("arena.zig").ReductionFrame; /// Call-by-need WHNF machine. Every suspended demand lives on the heap, and /// every update targets the original application (including shared redexes). /// Fuel counts application entries, not observations or continuation dispatch. /// Collection is opt-in: callers must register all externally held node IDs. pub fn reduce(root: u32, arena: *Arena, fuel: u64) ReduceError!u32 { + // Codecs repeatedly observe already-normal constructors. Do not borrow the + // continuation buffers for those calls, but never skip a due collection. + if (!arena.collectionDue() and arena.get(root).* != .app) return root; var remaining = fuel; - var frames: std.ArrayList(Frame) = .empty; - defer frames.deinit(arena.allocator); - var roots: std.ArrayList(u32) = .empty; - defer roots.deinit(arena.allocator); + // Reuse scratch capacity across codec/IO demands. Move it out while active; + // no stale redex IDs survive a return, including fuel/allocation failures. + var frames = arena.reduction_frames; + arena.reduction_frames = .empty; + defer { + frames.clearRetainingCapacity(); + arena.reduction_frames = frames; + } + var roots = arena.reduction_roots; + arena.reduction_roots = .empty; + defer { + roots.clearRetainingCapacity(); + arena.reduction_roots = roots; + } var current = root; while (true) { @@ -30,79 +39,142 @@ pub fn reduce(root: u32, arena: *Arena, fuel: u64) ReduceError!u32 { for (frames.items) |frame| try roots.append(arena.allocator, frame.redex); try arena.collect(roots.items); } + var frame: Frame = undefined; if (arena.get(current).* == .app) { if (remaining == 0) return error.FuelExhausted; remaining -= 1; - const func = arena.get(current).app.func; - try frames.append(arena.allocator, .{ .kind = .function, .redex = current }); - current = func; - continue; - } - const frame = frames.pop() orelse return current; - const orig = frame.redex; - if (frame.kind == .update) { - arena.get(orig).* = arena.get(current).*; - current = orig; - continue; + frame = .{ .kind = .function, .redex = current }; + current = arena.get(current).app.func; + } else { + frame = frames.pop() orelse return current; } - const app = arena.get(orig).app; - switch (frame.kind) { - .function => switch (arena.get(current).*) { - .leaf => { - arena.get(orig).* = .{ .stem = .{ .child = app.arg } }; - current = orig; - }, - .stem => |s| { - arena.get(orig).* = .{ .fork = .{ .left = s.child, .right = app.arg } }; - current = orig; - }, - .fork => |f| { - try frames.append(arena.allocator, .{ .kind = .discriminator, .redex = orig }); - current = f.left; - }, - .app => unreachable, - }, - .discriminator => { - const f = arena.get(app.func).fork; - switch (arena.get(current).*) { + + // Constructor demands need no suspension or extra machine-loop turn. + // Only an unevaluated application needs a continuation on the stack. + // Allocating rules always return to the safepoint before further work. + demand: while (arena.get(current).* != .app) { + const orig = frame.redex; + if (frame.kind == .update) { + arena.get(orig).* = arena.get(current).*; + current = orig; + break :demand; + } + const app = arena.get(orig).app; + switch (frame.kind) { + .function => switch (arena.get(current).*) { .leaf => { - try frames.append(arena.allocator, .{ .kind = .update, .redex = orig }); - current = f.right; - }, - .stem => |s| { - const a = try arena.alloc(.{ .app = .{ .func = s.child, .arg = app.arg } }); - const b = try arena.alloc(.{ .app = .{ .func = f.right, .arg = app.arg } }); - arena.get(orig).* = .{ .app = .{ .func = a, .arg = b } }; + arena.get(orig).* = .{ .stem = .{ .child = app.arg } }; current = orig; - }, - .fork => { - try frames.append(arena.allocator, .{ .kind = .argument, .redex = orig }); - current = app.arg; - }, - .app => unreachable, - } - }, - .argument => { - const f = arena.get(app.func).fork; - const branches = arena.get(f.left).fork; - switch (arena.get(current).*) { - .leaf => { - try frames.append(arena.allocator, .{ .kind = .update, .redex = orig }); - current = branches.left; + break :demand; }, .stem => |s| { - arena.get(orig).* = .{ .app = .{ .func = branches.right, .arg = s.child } }; + arena.get(orig).* = .{ .fork = .{ .left = s.child, .right = app.arg } }; current = orig; + break :demand; }, - .fork => |arg| { - const inner = try arena.alloc(.{ .app = .{ .func = f.right, .arg = arg.left } }); - arena.get(orig).* = .{ .app = .{ .func = inner, .arg = arg.right } }; - current = orig; + .fork => |f| { + frame.kind = .discriminator; + current = f.left; }, .app => unreachable, - } - }, - .update => unreachable, + }, + .discriminator => { + const f = arena.get(app.func).fork; + switch (arena.get(current).*) { + .leaf => { + frame.kind = .update; + current = f.right; + }, + .stem => |s| { + // The generated left application is demanded immediately. + // Fuse its constructor-only reduction when two further + // application entries fit in the caller's fuel budget. + if (remaining >= 2) { + switch (arena.get(s.child).*) { + .leaf => { + const b = try arena.allocApp(f.right, app.arg); + remaining -= 2; + arena.get(orig).* = .{ .fork = .{ .left = app.arg, .right = b } }; + current = orig; + break :demand; + }, + .stem => |k| if (arena.get(k.child).* == .leaf) { + remaining -= 2; + frame.kind = .update; + current = app.arg; + continue :demand; + }, + .fork => |k| if (arena.get(k.left).* == .leaf) { + // (K value arg) is the immediately demanded + // function of the outer application. Its + // private thunk need not be allocated. + const value = arena.get(k.right).*; + if (value == .fork and arena.get(value.fork.left).* == .leaf) { + remaining -= 2; + frame.kind = .update; + current = value.fork.right; + continue :demand; + } + const b = try arena.allocApp(f.right, app.arg); + switch (value) { + .leaf => { + remaining -= 2; + arena.get(orig).* = .{ .stem = .{ .child = b } }; + }, + .stem => |v| { + remaining -= 2; + arena.get(orig).* = .{ .fork = .{ .left = v.child, .right = b } }; + }, + else => { + remaining -= 1; + arena.get(orig).* = .{ .app = .{ .func = k.right, .arg = b } }; + }, + } + current = orig; + break :demand; + }, + else => {}, + } + } + const a = try arena.allocApp(s.child, app.arg); + const b = try arena.allocApp(f.right, app.arg); + arena.get(orig).* = .{ .app = .{ .func = a, .arg = b } }; + current = orig; + break :demand; + }, + .fork => { + frame.kind = .argument; + current = app.arg; + }, + .app => unreachable, + } + }, + .argument => { + const f = arena.get(app.func).fork; + const branches = arena.get(f.left).fork; + switch (arena.get(current).*) { + .leaf => { + frame.kind = .update; + current = branches.left; + }, + .stem => |s| { + arena.get(orig).* = .{ .app = .{ .func = branches.right, .arg = s.child } }; + current = orig; + break :demand; + }, + .fork => |arg| { + const inner = try arena.allocApp(f.right, arg.left); + arena.get(orig).* = .{ .app = .{ .func = inner, .arg = arg.right } }; + current = orig; + break :demand; + }, + .app => unreachable, + } + }, + .update => unreachable, + } + } else { + try frames.append(arena.allocator, frame); } } } diff --git a/ext/zig/src/reduce_test.zig b/ext/zig/src/reduce_test.zig @@ -99,7 +99,7 @@ test "non-moving collection preserves explicit roots and reuses dead slots" { for (0..1000) |_| _ = try fork(&a, t, t); const before = a.len(); try a.collect(&.{}); - try testing.expectEqual(@as(usize, 1000), a.free.items.len); + try testing.expectEqual(@as(usize, 0), a.free.items.len); for (0..1000) |_| _ = try stem(&a, t); try testing.expectEqual(before, a.len()); try testing.expectEqual(t, a.get(kept).stem.child); @@ -110,11 +110,16 @@ fn gcWorkload(allocator: std.mem.Allocator) !void { defer a.deinit(); const t = try a.alloc(.leaf); const k = try stem(&a, t); - const id = try fork(&a, try stem(&a, k), k); + const direct_id = try fork(&a, try stem(&a, k), k); + // S (K K) I is extensionally K, but not a constructor shortcut. Keep + // real distribution allocations here so GC/failure stress cannot disappear + // when the direct S K b identity is fused by the evaluator. + const extended_k = try fork(&a, try stem(&a, try fork(&a, t, k)), direct_id); + const id = try fork(&a, try stem(&a, extended_k), k); var x = t; for (0..100) |_| x = try app(&a, id, x); a.gc_enabled = true; - a.gc_interval = 8; + try a.setGcPolicy(8, 1); try testing.expect(a.get(try force(&a, x)).* == .leaf); try testing.expect(a.collections > 1); try a.collect(&.{x}); @@ -170,7 +175,7 @@ fn handleWorkload(allocator: std.mem.Allocator) !void { a.roots.clearRetainingCapacity(); try table.appendRoots(&a); try a.collect(&.{}); - try testing.expectEqual(@as(usize, 1000), a.free.items.len); + try testing.expectEqual(@as(usize, 0), a.free.items.len); try testing.expectEqual(@as(usize, 2), table.slots.items.len); } @@ -240,3 +245,331 @@ test "oversized host numbers return a decode failure rather than trapping" { const overflow = try fork(&a, try a.alloc(.leaf), max); try testing.expectEqual(@as(?u64, null), try codecs.toNumber(&a, overflow)); } + +test "headroom policy can shrink after a live peak and rejects zero atomically" { + var a = Arena.init(testing.allocator); + defer a.deinit(); + try a.setGcPolicy(8, 4); + var root = try a.alloc(.leaf); + for (0..100) |_| root = try stem(&a, root); + try a.collect(&.{root}); + try testing.expectEqual(@as(usize, 404), a.gc_interval); + try testing.expectError(error.InvalidData, a.setGcPolicy(0, 2)); + try testing.expectError(error.InvalidData, a.setGcPolicy(2, 0)); + try testing.expectEqual(@as(usize, 8), a.gc_min_interval); + try testing.expectEqual(@as(usize, 4), a.gc_live_multiplier); + try testing.expectEqual(@as(usize, 404), a.gc_interval); + a.get(root).* = .leaf; + try a.collect(&.{root}); + try testing.expectEqual(@as(usize, 8), a.gc_interval); + try a.setGcPolicy(1, std.math.maxInt(usize)); + try testing.expectEqual(std.math.maxInt(usize), a.gc_interval); + // Two live nodes exercise saturating multiplication, including on 32-bit. + const next = try stem(&a, root); + try a.collect(&.{next}); + try testing.expectEqual(std.math.maxInt(usize), a.gc_interval); +} + +test "collection trims only dead tails and reuses interior holes without moving roots" { + var a = Arena.init(testing.allocator); + defer a.deinit(); + const t = try a.alloc(.leaf); + const hole = try stem(&a, t); + const root = try fork(&a, t, t); + for (0..1000) |_| _ = try a.alloc(.leaf); + const capacity = a.nodes.capacity; + try a.collect(&.{root}); + try testing.expectEqual(root + 1, a.len()); + try testing.expectEqual(capacity, a.nodes.capacity); + try testing.expectEqualSlices(u32, &.{hole}, a.free.items); + try testing.expectEqual(hole, try a.alloc(.leaf)); + try testing.expectEqual(root + 1, try a.alloc(.leaf)); + try testing.expectEqual(t, a.get(root).fork.left); + try a.collect(&.{}); + try testing.expectEqual(@as(u32, 0), a.len()); + try testing.expectEqual(@as(usize, 0), a.free.items.len); + try testing.expectEqual(@as(u32, 0), try a.alloc(.leaf)); +} + +test "constructor fast return still honors collection with zero fuel" { + var a = Arena.init(testing.allocator); + defer a.deinit(); + const t = try a.alloc(.leaf); + const root = try stem(&a, t); + for (0..1000) |_| _ = try a.alloc(.leaf); + try a.setGcPolicy(1, 1); + a.gc_enabled = true; + try testing.expectEqual(root, try machine.reduce(root, &a, 0)); + try testing.expectEqual(@as(usize, 1), a.collections); + try testing.expectEqual(t, a.get(root).stem.child); + try testing.expectEqual(@as(u32, 2), a.len()); +} + +test "suspended demands preserve fuel boundaries and clear retained scratch on failure" { + var a = Arena.init(testing.allocator); + defer a.deinit(); + const t = try a.alloc(.leaf); + const delayed_function = try app(&a, t, t); + const root = try app(&a, delayed_function, t); + try testing.expectError(error.FuelExhausted, machine.reduce(root, &a, 1)); + try testing.expectEqual(@as(usize, 0), a.reduction_frames.items.len); + try testing.expectEqual(@as(usize, 0), a.reduction_roots.items.len); + try testing.expect(a.reduction_frames.capacity > 0); + _ = try machine.reduce(root, &a, 2); + try testing.expectEqual(t, a.get(root).fork.left); + // A delayed discriminator and a divergent, discarded argument. + const delayed_leaf = try app(&a, try fork(&a, t, t), t); + const poison = try app(&a, t, t); + a.get(poison).* = .{ .app = .{ .func = poison, .arg = t } }; + const selected = try app(&a, try fork(&a, delayed_leaf, t), poison); + _ = try machine.reduce(selected, &a, 2); + try testing.expect(a.get(selected).* == .leaf); + // A delayed selected result takes a second application entry. + const delayed_result = try app(&a, t, t); + const select = try app(&a, try fork(&a, t, delayed_result), poison); + try testing.expectError(error.FuelExhausted, machine.reduce(select, &a, 1)); + try testing.expectEqual(@as(usize, 0), a.reduction_frames.items.len); + _ = try machine.reduce(select, &a, 2); + try testing.expectEqual(t, a.get(select).stem.child); + // Delayed triage argument, then the stem arm creates another application. + const triage = try fork(&a, try fork(&a, t, t), t); + const argument = try app(&a, t, t); + const inspect = try app(&a, triage, argument); + try testing.expectError(error.FuelExhausted, machine.reduce(inspect, &a, 2)); + _ = try machine.reduce(inspect, &a, 1); + try testing.expectEqual(t, a.get(inspect).stem.child); +} + +test "collector allocation failure is retryable without losing live nodes or free slots" { + // Fail each allocation/growth in the first collection, then retry in the + // same arena. This also exercises dirty marks/grey left by failed marking. + for (0..12) |offset| { + var failing = testing.FailingAllocator.init(testing.allocator, .{}); + var a = Arena.init(failing.allocator()); + defer a.deinit(); + const t = try a.alloc(.leaf); + _ = try a.alloc(.leaf); // interior hole + var root = t; + // Traversing right leaves many left demands on the grey stack; + // each stem also shares t, exercising mark-on-enqueue deduplication. + for (0..100) |_| root = try fork(&a, try stem(&a, t), root); + for (0..100) |_| _ = try a.alloc(.leaf); + const before = a.len(); + failing.fail_index = failing.alloc_index + offset; + failing.resize_fail_index = failing.resize_index; + a.collect(&.{root}) catch |err| { + try testing.expectEqual(error.OutOfMemory, err); + try testing.expectEqual(before, a.len()); + try testing.expectEqual(@as(usize, 0), a.free.items.len); + try testing.expectEqual(@as(usize, 0), a.collections); + }; + failing.fail_index = std.math.maxInt(usize); + failing.resize_fail_index = std.math.maxInt(usize); + try a.collect(&.{root}); + try testing.expectEqual(@as(usize, 201), a.last_live); + try testing.expectEqualSlices(u32, &.{1}, a.free.items); + var cursor = root; + for (0..100) |_| { + try testing.expectEqual(t, a.get(a.get(cursor).fork.left).stem.child); + cursor = a.get(cursor).fork.right; + } + try testing.expectEqual(t, cursor); + } +} + +test "continuation allocation failure returns reusable scratch and evaluation can resume" { + var failing = testing.FailingAllocator.init(testing.allocator, .{}); + var a = Arena.init(failing.allocator()); + defer a.deinit(); + const t = try a.alloc(.leaf); + var root = t; + for (0..1000) |_| root = try app(&a, try fork(&a, t, root), t); + failing.fail_index = failing.alloc_index + 1; + failing.resize_fail_index = failing.resize_index; + try testing.expectError(error.OutOfMemory, force(&a, root)); + try testing.expectEqual(@as(usize, 0), a.reduction_frames.items.len); + try testing.expectEqual(@as(usize, 0), a.reduction_roots.items.len); + failing.fail_index = std.math.maxInt(usize); + failing.resize_fail_index = std.math.maxInt(usize); + try testing.expect(a.get(try force(&a, root)).* == .leaf); + try testing.expectEqual(@as(usize, 0), a.reduction_frames.items.len); +} + +// Small recursive oracle: the literal unfused rules, with one fuel charge for +// each application entry. Test inputs are shallow; this is not a runtime path. +fn unfused(root: u32, a: *Arena, fuel: *u64) machine.ReduceError!u32 { + const node = a.get(root).*; + if (node != .app) return root; + if (fuel.* == 0) return error.FuelExhausted; + fuel.* -= 1; + const function = a.get(try unfused(node.app.func, a, fuel)).*; + switch (function) { + .leaf => a.get(root).* = .{ .stem = .{ .child = node.app.arg } }, + .stem => |s| a.get(root).* = .{ .fork = .{ .left = s.child, .right = node.app.arg } }, + .fork => |f| { + const discriminator = a.get(try unfused(f.left, a, fuel)).*; + switch (discriminator) { + .leaf => { + const value = try unfused(f.right, a, fuel); + a.get(root).* = a.get(value).*; + }, + .stem => |s| { + const left = try app(a, s.child, node.app.arg); + const right = try app(a, f.right, node.app.arg); + a.get(root).* = .{ .app = .{ .func = left, .arg = right } }; + return unfused(root, a, fuel); + }, + .fork => |branches| { + const argument = a.get(try unfused(node.app.arg, a, fuel)).*; + switch (argument) { + .leaf => { + const value = try unfused(branches.left, a, fuel); + a.get(root).* = a.get(value).*; + }, + .stem => |s| { + a.get(root).* = .{ .app = .{ .func = branches.right, .arg = s.child } }; + return unfused(root, a, fuel); + }, + .fork => |v| { + const inner = try app(a, f.right, v.left); + a.get(root).* = .{ .app = .{ .func = inner, .arg = v.right } }; + return unfused(root, a, fuel); + }, + .app => unreachable, + } + }, + .app => unreachable, + } + }, + .app => unreachable, + } + return root; +} + +fn equalUnfused(a: *Arena, x: u32, b: *Arena, y: u32) !void { + var fuel: u64 = 10000; + const expected = a.get(try unfused(x, a, &fuel)).*; + const actual = b.get(try machine.reduce(y, b, 10000)).*; + try testing.expectEqual(std.meta.activeTag(expected), std.meta.activeTag(actual)); + switch (expected) { + .leaf => {}, + .stem => |s| try equalUnfused(a, s.child, b, actual.stem.child), + .fork => |f| { + try equalUnfused(a, f.left, b, actual.fork.left); + try equalUnfused(a, f.right, b, actual.fork.right); + }, + .app => unreachable, + } +} + +fn fusionFixture(a: *Arena, variant: usize, argument: usize) !u32 { + const t = try a.alloc(.leaf); + const k = try stem(a, t); + const payload = try fork(a, k, t); + const value = switch (variant) { + 0 => t, + 1 => k, + 2 => try fork(a, t, t), + 3 => try fork(a, t, k), + 4 => try fork(a, t, try fork(a, t, payload)), + 5 => try fork(a, t, try app(a, try fork(a, t, k), t)), + // Negative controls: delayed discriminator, non-K stem, non-K fork. + 6 => try fork(a, try app(a, try fork(a, t, t), t), k), + 7 => try stem(a, k), + 8 => try fork(a, t, try fork(a, k, t)), + else => unreachable, + }; + const x = switch (argument) { + 0 => t, + 1 => k, + 2 => payload, + 3 => try app(a, try fork(a, t, payload), t), + else => unreachable, + }; + return app(a, try fork(a, try stem(a, value), t), x); +} + +test "fused distribution agrees with literal rules at every small fuel boundary" { + for (0..9) |variant| for (0..4) |argument| for (0..20) |budget| { + var expected = Arena.init(testing.allocator); + defer expected.deinit(); + var actual = Arena.init(testing.allocator); + defer actual.deinit(); + const x = try fusionFixture(&expected, variant, argument); + const y = try fusionFixture(&actual, variant, argument); + var fuel: u64 = budget; + const reference_result = unfused(x, &expected, &fuel); + if (reference_result) |_| { + _ = try machine.reduce(y, &actual, budget); + try equalUnfused(&expected, x, &actual, y); + } else |err| { + try testing.expectEqual(error.FuelExhausted, err); + try testing.expectError(error.FuelExhausted, machine.reduce(y, &actual, budget)); + // Partial graph shapes can differ; both must remain resumable. + try equalUnfused(&expected, x, &actual, y); + } + }; +} + +test "fused identity and constant selection discard divergent branches" { + var a = Arena.init(testing.allocator); + defer a.deinit(); + const t = try a.alloc(.leaf); + const poison = try app(&a, t, t); + a.get(poison).* = .{ .app = .{ .func = poison, .arg = poison } }; + const k = try stem(&a, t); + const identity = try fork(&a, try stem(&a, k), poison); + const result = try app(&a, identity, t); + const before = a.allocations; + _ = try machine.reduce(result, &a, 3); + try testing.expect(a.get(result).* == .leaf); + try testing.expectEqual(before, a.allocations); + const constant = try fork(&a, try stem(&a, try fork(&a, t, try fork(&a, t, k))), poison); + const discard = try app(&a, constant, poison); + const allocated = a.allocations; + _ = try machine.reduce(discard, &a, 3); + try testing.expectEqual(t, a.get(discard).stem.child); + try testing.expectEqual(allocated, a.allocations); + // Selecting divergence must still demand it, rather than return a thunk. + const demand = try app(&a, identity, poison); + try testing.expectError(error.FuelExhausted, machine.reduce(demand, &a, 100)); +} + +test "fused selection updates shared demanded payloads and survives collection" { + var a = Arena.init(testing.allocator); + defer a.deinit(); + const t = try a.alloc(.leaf); + const payload = try app(&a, try fork(&a, t, t), t); + const poison = try app(&a, t, t); + a.get(poison).* = .{ .app = .{ .func = poison, .arg = poison } }; + const constant = try fork(&a, t, try fork(&a, t, payload)); + const root = try app(&a, try fork(&a, try stem(&a, constant), poison), poison); + try a.roots.append(a.allocator, payload); // independently retained alias + a.gc_enabled = true; + try a.setGcPolicy(1, 1); + _ = try machine.reduce(root, &a, 4); + try testing.expect(a.get(root).* == .leaf); + try testing.expect(a.get(payload).* == .leaf); + try a.collect(&.{root}); + try testing.expect(a.get(root).* == .leaf); + try testing.expect(a.get(payload).* == .leaf); +} + +test "scalar application allocation initializes dirty recycled slots completely" { + var a = Arena.init(testing.allocator); + defer a.deinit(); + const t = try a.alloc(.leaf); + const hole = try app(&a, t, t); + a.get(hole).* = .{ .app = .{ .func = hole, .arg = hole } }; + const kept = try stem(&a, t); + try a.collect(&.{kept}); + try testing.expectEqual(hole, try a.allocApp(kept, t)); + try testing.expectEqual(kept, a.get(hole).app.func); + try testing.expectEqual(t, a.get(hole).app.arg); + _ = try machine.reduce(hole, &a, 1); + try testing.expectEqual(t, a.get(hole).fork.left); + try testing.expectEqual(t, a.get(hole).fork.right); + try a.collect(&.{kept}); + try testing.expectEqual(hole, try a.alloc(.leaf)); + try testing.expect(a.get(hole).* == .leaf); +} diff --git a/ext/zig/src/source_text.zig b/ext/zig/src/source_text.zig @@ -82,7 +82,7 @@ test "source scalar codec preserves exact UTF-8 including NUL, CRLF and nonchara try std.testing.expectEqual(expected.len, list.items.len); for (list.items, expected) |item, n| try std.testing.expectEqual(n, (try codecs.toNumber(&arena, item)).?); arena.gc_enabled = true; - arena.gc_interval = 1; + try arena.setGcPolicy(1, 1); const result = try toUtf8(&arena, scalars); defer arena.allocator.free(result); try std.testing.expectEqualStrings(bytes, result); @@ -123,7 +123,7 @@ test "source scalar codec accepts lazy scalar numerals" { const delayed = try arena.alloc(.{ .app = .{ .func = constant, .arg = leaf } }); const list = try codecs.ofList(&arena, &.{delayed}); arena.gc_enabled = true; - arena.gc_interval = 1; + try arena.setGcPolicy(1, 1); const bytes = try toUtf8(&arena, list); defer arena.allocator.free(bytes); try std.testing.expectEqualStrings("😀", bytes); diff --git a/ext/zig/tests/handles_test.c b/ext/zig/tests/handles_test.c @@ -9,6 +9,9 @@ int main(void) { assert(arb_abi_version() == 2); arb_ctx_t *ctx = arboricx_init(), *other = arboricx_init(); assert(ctx && other); + assert(arb_set_gc_policy(ctx, 0, 1) == ARB_INVALID_DATA); + assert(arb_set_gc_policy(ctx, 1, 0) == ARB_INVALID_DATA); + OK(arb_set_gc_policy(ctx, 1, 1)); /* stress safepoints, including codecs */ arb_handle_t leaf, alias, stem, child, app, result; OK(arb_leaf(ctx, &leaf)); OK(arb_retain(ctx, leaf, &alias)); diff --git a/test/bootstrap_generations.py b/test/bootstrap_generations.py @@ -85,6 +85,8 @@ def main(): parser.add_argument('--tricu', default='result/bin/tricu') parser.add_argument('--execution-host', choices=('haskell', 'zig'), default='haskell') parser.add_argument('--zig-library', type=Path, help='explicit Nix-built owned-ABI library; required for Zig execution') + parser.add_argument('--zig-gc-policy', type=int, nargs=2, metavar=('MIN_ALLOCATIONS', 'LIVE_MULTIPLIER'), + help='explicit positive u32 allocation headroom (not an RSS limit); Zig only') parser.add_argument('--backend', choices=BACKENDS, default=DEFAULT_BACKEND, help='generation lowering backend (default: %(default)s); never inferred from previous') parser.add_argument('--worker-bundle', type=Path, help='explicit pinned transition worker exporting the selected backend entrypoint; no fallback') @@ -109,6 +111,9 @@ def main(): parser.error('--zig-library is required exactly when --execution-host zig is selected') if args.zig_library and not args.zig_library.is_file(): parser.error('selected Zig library is unavailable; no fallback') + if args.zig_gc_policy and (args.execution_host != 'zig' or + any(not 0 < n <= 0xffffffff for n in args.zig_gc_policy)): + parser.error('--zig-gc-policy requires Zig and two positive u32 values') previous = args.previous.resolve() if args.previous else None output = args.output.resolve() from bootstrap_retention_lock import protect_generations @@ -224,6 +229,8 @@ def main(): transport_runtime_sha256=hashlib.sha256(Path(tricu).read_bytes()).hexdigest(), executor_sha256=hashlib.sha256((REPO / 'test/zig_generation_executor.py').read_bytes()).hexdigest() if args.zig_library else None, **inputs) + if args.zig_gc_policy: + provenance['zig_gc_policy'] = args.zig_gc_policy (output / 'inputs.json').write_text(json.dumps(provenance, indent=2) + '\n') (output / 'source-map.v1').write_text(map_text) cmd = [tricu, 'arboricx', 'run', '--store', str(output / 'compiler-cas'), @@ -243,6 +250,8 @@ def main(): cmd = [sys.executable, str(REPO / 'test/zig_generation_executor.py'), '--library', str(args.zig_library.resolve()), '--output', str(output), '--memory-mib', str(args.memory_mib), '--seconds', str(math.ceil(remaining()))] + if args.zig_gc_policy: + cmd += ['--gc-policy', *map(str, args.zig_gc_policy)] (output / 'invocation.json').write_text(json.dumps(cmd, indent=2)) log = output / f'native-{time.time_ns()}.log' (output / 'status.json').write_text(json.dumps(dict(complete=False, diff --git a/test/cross_host_generation.py b/test/cross_host_generation.py @@ -19,8 +19,11 @@ def main(): p.add_argument('--tricu', type=Path, required=True) p.add_argument('--library', type=Path, required=True) p.add_argument('--directory', type=Path, required=True) + p.add_argument('--zig-gc-policy', type=int, nargs=2, metavar=('MIN_ALLOCATIONS', 'LIVE_MULTIPLIER')) p.add_argument('--compatibility', action='store_true', help='also check broader syntax, invalidation and negative controls') args = p.parse_args() + if args.zig_gc_policy and any(not 0 < n <= 0xffffffff for n in args.zig_gc_policy): + p.error('GC policy requires positive u32 values') directory = args.directory.resolve() directory.mkdir(parents=True, exist_ok=False) source = directory / 'source' @@ -66,12 +69,18 @@ def main(): '--execution-host',host,'--source-root',source,'--map',source_map,'--root','app', '--backend','stable','--memory-mib','1024','--wall-seconds','120', '--initial-seconds','60','--extended-seconds','100'] - if host == 'zig': command += ['--zig-library',args.library.resolve()] + if host == 'zig': + command += ['--zig-library',args.library.resolve()] + if args.zig_gc_policy: + command += ['--zig-gc-policy', *map(str, args.zig_gc_policy)] commands[host] = command run(host+'-cold',command+['--fresh']) status = json.loads((output/'status.json').read_text()) assert status['complete'] and status['execution_host']==host and status['fresh_store'] assert status['transport_host']=='haskell' + if host == 'zig' and args.zig_gc_policy: + assert status['zig_gc_policy'] == args.zig_gc_policy + assert json.loads((output/'execution.json').read_text())['gc_policy'] == args.zig_gc_policy runtime = args.tricu if host == 'haskell' else args.library assert status['runtime_sha256']==hashlib.sha256(runtime.read_bytes()).hexdigest() invocation = json.loads((output/'invocation.json').read_text()) diff --git a/test/zig_generation_executor.py b/test/zig_generation_executor.py @@ -67,6 +67,9 @@ def child(args): request = json.loads((run / 'execution-request.json').read_text()) started = time.monotonic() try: + if args.gc_policy: + configure = bind('arb_set_gc_policy', [ctx_t, C.c_uint32, C.c_uint32]) + check(configure(ctx, *args.gc_policy)) compiler_bytes = (run / 'compiler.arboricx').read_bytes() adapter_bytes = (run / 'io-adapter.arboricx').read_bytes() if hashlib.sha256(compiler_bytes).hexdigest() != request['compiler_bundle_sha256'] or hashlib.sha256(adapter_bytes).hexdigest() != request['adapter_sha256']: @@ -111,6 +114,8 @@ def child(args): record = dict(ok=bool(ok.value), diagnostic=diagnostic, prepare_seconds=prepared-started, initial_reduce_seconds=forced-prepared, io_seconds=ran-forced, total_seconds=time.monotonic()-started) + if args.gc_policy: + record['gc_policy'] = args.gc_policy save(run / 'execution.json', record) if not ok.value: raise RuntimeError(diagnostic) @@ -125,8 +130,11 @@ if __name__ == '__main__': parser.add_argument('--library', type=Path, required=True) parser.add_argument('--output', type=Path, required=True) parser.add_argument('--memory-mib', type=int, required=True) + parser.add_argument('--gc-policy', type=int, nargs=2, metavar=('MIN_ALLOCATIONS', 'LIVE_MULTIPLIER')) parser.add_argument('--seconds', type=int, required=True) args = parser.parse_args() if not 0 < args.memory_mib <= 65536 or not 0 < args.seconds <= 7200: parser.error('invalid resource boundaries') + if args.gc_policy and any(not 0 < n <= 0xffffffff for n in args.gc_policy): + parser.error('GC policy requires positive u32 values') child(args)