SCP lowering experiments

This history collects the experiments that led to the lexical compiler. Current semantics live in the language contract; entry points and focused commands live in the usage guide. Measurements below are historical snapshots of the implementation at each step.

From authored trees to reusable programs

The first interpreter accepted initial state and ordered input/state rules as ordinary trees. FizzBuzz exercised countdown state, dynamic worker selection, and reversible replacement of designated output literals. A located-token layout specimen then reused the existing parser reply encodings, including remainder, location, consumption, commitment, attempt and alternative behavior. The layout specimen used a small name parser after a fixed x = prefix.

The source frontend first lowered FizzBuzz, then a registry-specific route DSL. The registry trial handled already-parsed (method, target) requests, health responses, local bundle reads and fallback replies. Its concise source exposed route order and retained state, but required a dedicated vocabulary and runner. That motivated the uniform base frontend: authored map, filter, fold, append, reverse, Option and prefix/path consumers shared structural patterns and generic call, recurrence and delegation.

The generic effect driver separated application operations from named receivers. A program returns Done value or Await operation argument receiver environment. The driver resolves the receiver before invoking the handler and carries that resolved executable across IO. Success/error resumes with explicit retained state. Tests include successive effects with different receivers and confirm a missing receiver prevents its handler from running. Registry fetch and text-summary clients share this protocol while selecting their own handlers and presentation. The summary counts LF-separated lines, including a partial final line, and maps read failures to its CLI status.

Early fixtures embedded SCP bytes through Python. Later pinned-source linking staged unchanged bytes under arboricx.scp-source.v1 and an explicit arboricx.scp-imports.v1 descriptor. Verified CAS reads checked source hashes and declared exports before compilation; separate manifests exposed original IR. Native project composition subsequently moved to the shared source graph and build-driver, described in the usage guide.

Lowering progression

Implementation	Work moved into preparation	Remaining runtime structure
Interpreter	Authored source to IR	Rule/pattern/output dispatch, named capture lookup, Results
Prepared patterns	Pattern tags to matcher closures	Output interpretation and capture associations
Fully prepared	Output tags to closures	Rule selection, capture associations, Results, dynamic calls
Direct	Ordered rules to executable branches	Capture associations and checked output helpers
Indexed / bound	Capture names to positions; known state substituted; literal workers linked	Capture-value lists and Result plumbing
Projection	Captures to input/state accessors	Context pairs, repeated projections, Result plumbing
Success islands	Safe constructor subtrees to plain value builders with one outer ok	Checked calls/control and capture-value lists
CPS	Success/failure continuations through calls and outputs	Indexed matcher lists and Maybe replies
Lexical	Structural matching to core case branches and lexical binders	Tree reduction, explicit state, continuations and dynamic worker calls

Known-state binding rejects recurrence in retained templates, because recurrence can change the state used for specialization. Each experimental runner requires its matching executable representation. The lexical ABI, worker descriptors and validated authored-IR boundary are documented with the implementation and tests. Original IR is exported separately for inspection and transformation.

Projection recomputes prefix suffixes when referenced and can retain large input contexts. Its measurements showed no additional speedup over indexed lowering. Success islands preserved error order, laziness and return-slot shadowing, with a small timing improvement. CPS alone left the capture-list matcher cost in place. These alternatives remain explicit comparison modules under lib/consumer/experiments/lowerings/ (consumers.lowerings.*). The lexical compiler removed that machinery by lowering captures to lexical variables and ordered patterns to direct branches.

Nested-pattern regression

An early packed FizzBuzz worker failed at the third element although a concrete source-eval probe passed. Matching the left child of a Fork introduced binders; the saved right-child expression then referred to the wrong de Bruijn index. lContext tracks compile-time depth and lRaise shifts the sibling expression across new binders. The fresh-store FizzBuzz test checks the third element, complete output and agreement with independently loaded IR.

The initial eta backend exceeded a 165-second FizzBuzz compilation budget. The fixture explicitly selected sourceCompileCoreKiselyov through lCompileWith, using the same structural code generator. This illustrates why compiler/export probes and concrete execution checks both matter.

Historical measurements

Preloaded intervals begin after packed exports load. They include input construction, result forcing/equality and marker output; file workloads also include IO and presentation. Compilation, packing, import and loading are excluded. Rows from different experiments are separate runs. Process-wide RSS includes library/loading costs; allocation and memory scaling require separate measurements.

Pure list and state workloads

500-item map + filter	SCP median	Lambda median	Executable nodes / bytes
Interpreter	271.6 ms	6.9 ms	1,329 / 16,047
Prepared patterns	178 ms	6–7 ms	1,121 / 13,539
Fully prepared	82 ms	6–7 ms	713 / 8,871
Indexed	32.1 ms	6.4 ms	603 / 7,636
Projection	32.3 ms	6.5 ms	606 / 7,679
Success islands	31.5 ms	6.4 ms	610 / 7,715
CPS	34.9 ms	6.3 ms	—
Lexical	12.0 ms	6.2 ms	525 / 6,670

The lambda map/filter root was 133 nodes / 1,945 bytes. Paired success/indexed runs measured 30.4/33.6 ms and 30.8/32.5 ms at 500 items, and 60.9/67.6 ms at 1,000, with overlapping samples and order effects.

Lexical map and filter separately measured 8.7/5.8 ms and 8.7/5.7 ms at 500 items. The 1,000-item composition measured 23.3/14.8 ms. Lexical FizzBuzz measured 22.8/11.5 ms at 500 and 45.0/23.3 ms at 1,000; its executable root was 676 nodes / 8,477 bytes versus lambda's 286 / 3,734. Adding its authored IR sidecar produced 736 nodes / 9,325 bytes.

Staging reduced interpreter overhead substantially on these workloads. The lexical implementation improved further while executable size remained larger than the lambda baseline. Early frontend-inclusive and fresh-process loader probes were dominated by compilation/startup; later preloaded probes replaced those comparisons.

Whole clients

Workload	SCP median	Lambda median
Prepared registry, 20 local reads	175.0 ms	41.3 ms
Direct registry, balanced paired run, 20 reads	162.3 ms	42.2 ms
Linked registry, 20 reads	142.6 ms	41.3 ms
Projected registry, 20 reads	149.1 ms	41.8 ms
Interpreted/prepared summary, 20 reads	209.8 ms	27.7 ms
Lexical summary, 20 reads	52.1 ms	28.5 ms

The direct registry run also measured prepared execution at 169.7 ms. Within that process, prepared-minus-direct differences ranged from -13.3 to +22.5 ms, with median 8.7 ms. Linked's paired differences against prepared ranged from -12.4 to +47.7 ms, with median 34.5 ms. Whole-fetch improvements were noisy. Pure linked routing measured 75.9/54.9 ms for 50 executions; post-read applied receivers measured 15.9/14.6 ms. Named dispatch added visible overhead.

The summary used a 30-line, 180-byte file. Independently packed lexical and lambda roots were 1,276 nodes / 15,620 bytes and 286 / 3,709. The lexical behavior gate included three IR exports, totaling 1,352 / 16,805. Its open-state entry and receiver also accepted a different compiled counter at runtime and returned the substituted result without recompiling either receiver.

Retained coverage and research questions

The focused interpreter/lexical differential gate compares authored map/filter composition, reverse, prefix/path handling and literal replacement. Named parse, compile, agreement and expected-value assertions identify failures without collapsing failed Results into leaf. It packs the lexical executable with IR sidecars and runs it from a fresh store; obsolete lowerings are not part of this gate. Their standalone pattern specimens remain separate research probes. The lexical gate adds validation, worker ABI, delegation, prefix elaboration, authored source, named effects and generated reference/compiler differential cases. FizzBuzz and layout retain distinct nested-pattern/parser regressions. The source-link and text-summary gates cover verified source composition, tampering, real reads, missing files and worker injection.

Prioritize compiler cost, artifact size, memory growth, diagnostics and source navigation on practical multi-module programs. Measure source compilation and deployed execution separately. Use a new lowering only when a real workload exposes a cost worth addressing, preserving explicit state/effects, exact data encodings and inspectable authored IR.