tricu/docs/module-system-design.md

Module System and Content Store Design

Status: design draft.

This document records the intended direction for reworking tricu modules, imports, Arboricx storage/transport, and the content store. It is not an implementation plan yet; it is a shared design target.

1. Problem Statement

The current module/import/content-store system is useful as a prototype, but it is not coherent enough to build on indefinitely.

Current behavior combines several partially-overlapping systems:

• !import "path.tri" Namespace and !import "path.tri" !Local perform filesystem-relative source preprocessing;
• imported definitions are flattened into one program;
• namespace qualification is implemented by string rewriting;
• evaluation uses a flat Map String T environment;
• the Haskell content store stores Tree Calculus Merkle nodes plus an ad hoc terms table with comma-separated names and tags;
• the REPL can resolve names from the content store, including multiple versions;
• Arboricx bundles provide compact indexed transport objects;
• lib/arboricx/server.tri already sketches a filesystem-backed object store.

This works only when users and maintainers are mindful of sharp edges:

• names serve too many roles at once;
• modules are not first-class semantic objects;
• imports are closer to AST paste-and-prefix than resolution;
• !Local imports can create global collisions;
• content identity, human aliases, source files, and evaluated terms are not cleanly separated;
• the SQLite schema is convenient but not a principled content-addressed store;
• Arboricx transport and long-lived storage are not clearly distinguished.

2. Design Principles

2.1 Content addressability is foundational

Immutable content should be identified by hashes. Human names should be metadata or workspace aliases over content, not semantic identity.

This follows the core lesson from systems such as Unison: separate stable content identity from ergonomic naming and namespace organization.

2.2 The content store is language-neutral

The content store must not be married to tricu or Haskell.

It stores a small set of portable Arboricx artifacts: module manifests, complete tree terms, and direct Contract terms. Lower-level Merkle/bundle formats exist for transport and DAG tooling, but the store core should treat all objects as content-addressed bytes with formats/media types.

tricu and Haskell are clients/tooling. They are not the semantic owners of the store.

2.3 Contracts are portable enough to integrate

The store may integrate with Contracts because a contract is itself an ordinary Tree Calculus term. Contracts are not a Haskell-private or tricu-private semantic layer.

A module manifest may reference a contract object, but the contract is evaluated by ordinary Tree Calculus reduction, not by a special checker.

2.4 Modules should reflect definitions as they actually exist

The module system should conform to the reality of content-addressed immutable artifacts and mutable human aliases. We should not contort definitions to fit a traditional text-file module system if that fights the storage model.

2.5 Transport and storage are different jobs

Indexed Arboricx bundles are excellent transport/execution objects. Merkle DAGs are better long-lived persistence objects. These should remain separate but interoperable representations.

3. Conceptual Architecture

Content Store
  neutral content-addressed object store

Arboricx CAS / Merkle Store
  Tree Calculus node/object formats suitable for persistence and dedupe

Arboricx Bundle
  compact indexed transport/execution format

Contract Term
  ordinary tree-valued contract function, applied by reduction

Module Manifest
  immutable export map from names to content objects and optional contract terms

Workspace
  mutable aliases, selected versions, package pins, and user-facing names

tricu
  one frontend/toolchain that emits/consumes these portable artifacts

The content store stores objects. Arboricx defines important object formats. Contracts are ordinary tree-valued functions; tricu produces and consumes those formats.

3.1 Execution imports versus contract checking

Import resolution has two intentionally different performance profiles.

For normal execution/evaluation, resolving a module import should hydrate only the executable exports directly demanded by the importing source. Exported Tree Calculus values are complete normal forms: importing foo does not require hydrating separate bar or baz exports that may have helped build it. This is the fast path for !import, including !Local imports.

Contract checking is a runtime boundary check. It may load exported contract term objects for the symbols that participate in a boundary. That slower path remains separate from execution hydration:

Haskell loads contract terms.
tricu applies them by reduction.

Reusable contract catalogs are ordinary tricu libraries/tree terms, not a separate core CAS artifact kind.

For locally built workspace modules, advertised direct export contracts may be checked before the manifest alias is written. Producer checking includes advertised contracts from any imported modules used by that source, so a module cannot publish a local annotated export that contradicts a dependency's exported contract. If producer checking fails, the module alias is not written.

Consumer checking then resolves selected module exports and loads their exported contract term objects. Those contracts are applied at the import boundary using the standard contract helpers. For external or prebuilt manifests, the advertised contract is a trusted boundary declaration; the consumer may still re-apply it at the boundary.

4. Content Store Direction

4.1 Store core

The store core should be a content-addressed object store:

hash -> object bytes
hash -> object kind / media type
hash -> optional metadata/index entries

The hash should be over canonical bytes with domain separation. The object kind or media type determines how a client interprets those bytes.

Current module/check object kinds:

arboricx.module-manifest.v1
arboricx.tree-term.v1
arboricx.tree-term.v1

Merkle nodes and indexed bundles remain lower-level Arboricx transport/DAG formats, but they are not the module/eval storage model. typed programs and contract catalogs are ordinary tree terms unless a future external tooling use case proves that they need their own object kind.

The store core should not need to know what a tricu definition means.

4.2 Filesystem-backed layout

The long-term store should converge with the direction already sketched in lib/arboricx/server.tri:

store/
  objects/
    abc/
      abc123...object
  aliases/
    names/
    modules/
    packages/
  manifests/
  tmp/

SQLite may remain useful as an optional index/cache, but it should not be the canonical store model.

4.3 Structural references, not language dependencies

The store may understand structural content references when they are part of an object format. For example, a Merkle node naturally references child hashes:

Leaf
Stem childHash
Fork leftHash rightHash

This is not a tricu dependency graph. It is content structure.

Language/tool-level relationships such as "compiled from source", "exported by module", or "checked with contract" can live in manifests or indexes. They should not be required by the store core.

5. Arboricx Role

Arboricx should be understood as a family of portable Tree Calculus artifact formats, not as a single storage mechanism.

5.1 Arboricx Bundle

The existing indexed .arboricx format remains the preferred transport and execution object:

• compact;
• self-contained;
• deterministic;
• easy to parse in constrained runtimes;
• suitable for deployment and HTTP serving;
• structurally verifiable without hash recomputation per node.

It says:

Here is everything you need, densely packed.

5.2 Arboricx CAS / Merkle Store

The persistent store should use content-addressed structural objects:

Leaf
Stem childHash
Fork leftHash rightHash

This enables dedupe across definitions, modules, packages, and versions. A large program that shares subtrees with other programs should not store those subtrees multiple times.

It says:

Here are immutable objects, addressable independently.

5.3 Pack and unpack

Transport and storage should interoperate explicitly:

CAS root(s) -> pack   -> indexed Arboricx bundle
Arboricx bundle -> unpack -> CAS root(s)

The bundle can be treated as an opaque content-addressed blob by the store, and it can also be unpacked into Merkle nodes for dedupe and partial reuse.

6. Modules

6.1 Module identity

A module should be an immutable manifest object. Its identity is the hash of its canonical manifest bytes.

A module name is not identity. It is a workspace alias or package-level alias to a module hash.

6.2 Module contents

A module manifest should primarily be an export map:

module hash
exports:
  name -> content reference
metadata:
  package
  version
  description
  license
  createdBy
optional:
  contract artifact refs
  ABI/media type info
  source/provenance refs

The manifest should be portable and mostly format-oriented. It should not depend on Haskell data structures or tricu-specific internal semantics.

6.3 Export entries

An export entry may eventually look conceptually like:

name: "map"
object: sha256:...
kind: arboricx.tree-term.v1
abi: arboricx.abi.tree.v1
contract: sha256:...       -- optional contract term
source: sha256:...     -- optional source/provenance object

Executable module exports are complete normalized tree terms stored as one arboricx.tree-term.v1 object per named export. Merkle-node storage remains available for DAG-oriented tooling, but module/eval imports should not store or hydrate every subtree as a separate filesystem object.

6.4 Import behavior

Imports should resolve module aliases or content references to module manifests, then bind selected exports into the local source scope.

Export selection has one intentional aggregator special case:

module with local top-level definitions -> exports only those local definitions
module with only imports                -> reexports the evaluated import env

This lets files such as prelude.tri act as explicit barrel modules without making every ordinary module reexport its imports. A module that defines even one local top-level name does not implicitly reexport imported names.

The future pipeline should be:

parse source
resolve imports/names to module exports and content refs
lower source using resolved refs
emit a contract artifact
check evidence when requested
store/export artifacts

It should not be:

paste imported ASTs into one file and rewrite strings

7. Workspace Layer

Mutable human-facing state belongs in a workspace layer.

Examples:

List        -> module hash
Http        -> module hash
map         -> definition/tree hash
selected List version -> module hash
package pin prelude -> package/module hash

The workspace is where names, selections, pins, and aliases live. Renaming should usually mutate workspace aliases, not immutable content objects.

This gives humans stable ergonomic names without making names semantic identity.

8. Definition Identity

There are two useful identities and we should support both.

8.1 Tree identity

A Tree Calculus value has a Merkle root hash. This identifies the executable tree itself.

This is the right identity for:

• execution;
• dedupe;
• bundle roots;
• low-level artifact sharing.

8.2 Module/export identity

The module manifest is the higher-level artifact boundary. It pairs each export name with its compiled tree term and an optional contract term reference.

The content store should not require extra definition/source/provenance objects, and fully untyped Tree Calculus code must remain valid.

9. Contract Integration

Contracts attach to modules/exports as ordinary tree-term objects. A contract is a tricu function Tree -> Result Tree Tree; it is not a special artifact kind and it does not require a separate checker binary.

A module manifest pairs each export name with its compiled tree term and an optional contract term reference. The importer loads the contract object and applies it at the boundary.

An imported definition can be assigned a local name while lowering source. Its global identity remains a content hash or module export ref. The intended split is:

Local source name: "List.map"
Resolved object:    sha256:...
Exported contract:  sha256:...

There is no typed-program evidence graph and no local numeric checker symbols. The contract term itself is the authority.

Untyped code remains valid with no contract artifact. If a boundary has no contract information, it simply performs no runtime check. We should not pretend that untyped functions have an implicit Any -> Any -> ... contract.

10. Import Syntax Direction

Exact syntax is future work, but the current !import form should be considered a transitional mechanism.

Future imports should distinguish:

• path-based source imports for local development;
• workspace/module alias imports;
• explicit content-addressed imports;
• selected/exposed names;
• qualified versus unqualified binding.

Possible directions:

import "./list.tri" as List
import List exposing (map foldl)
import #abc123... as List

The syntax should be designed after the object/module model is clearer.

11. Migration Strategy

A plausible migration path:

1. Define the neutral object store model and filesystem layout.
2. Implement Merkle node persistence against that layout.
3. Add pack/unpack between CAS roots and indexed Arboricx bundles.
4. Replace ad hoc SQLite terms names/tags with workspace aliases or a clearer index layer.
5. Define module manifest objects.
6. Teach source imports to resolve manifests/exports instead of rewriting ASTs.
7. Attach contract terms to module exports.
8. Gradually migrate existing lib/ and demos/ imports.

Compatibility shims may keep existing !import working during migration.

12. Open Questions

• What exact canonical byte format should store objects use?
• Should module manifests be binary, tree-encoded, or both?
• What media type/kind registry do we need first?
• How should object references be represented in source syntax?
• How should workspaces be stored and shared?
• What is the minimum useful module manifest?
• Should source files compile directly to module manifests, or should manifests be produced by explicit package commands?
• How much Arboricx bundle metadata should reference CAS roots?
• What GC/reachability model should the store eventually use?

13. Summary

The desired design is:

Content store:
  portable CAS for immutable objects and structural references

Arboricx bundle:
  compact indexed transport/execution object

Arboricx CAS:
  persistent Merkle DAG/object representation for dedupe and partial reuse

Modules:
  immutable manifests mapping export names to content objects and optional
  contracts

Workspace:
  mutable human aliases, version selections, and package/module pins

Contracts:
  ordinary tree-valued functions attached to exports and applied by pure Tree
  Calculus reduction at boundaries

The key architectural rule is that hashes provide stable identity, while names provide human usability. The module system should be built on that separation.