tricu/docs/content-store-and-module-format.md

# Content Store and Module Format Design

Status: concrete design draft.

This document narrows the higher-level module-system direction into concrete
format and storage decisions. It intentionally avoids source/provenance details:
modules export usable portable artifacts, not edit history.

Related design overview: `docs/module-system-design.md`.

## 1. Scope

This document specifies the first target shape for:

- a neutral filesystem-backed content-addressed store;
- Arboricx Merkle node persistence;
- indexed Arboricx bundle import/export as transport;
- module manifests as immutable export maps;
- workspace aliases as mutable human-facing references;
- Contract artifact attachment to module exports.

It does not specify:

- package manager semantics;
- dependency solving;
- source-level rebuild/provenance metadata;
- final import syntax;
- garbage collection;
- registry/sync protocol.

## 2. Non-Negotiable Boundaries

The content store is not `tricu`-specific and is not Haskell-specific.

The store may contain objects produced by `tricu`, Haskell, Tree Calculus tools,
Arboricx tooling, or future frontends. The store core only knows object bytes,
object kinds, hashes, aliases, and optionally structural references for known
portable formats.

Contracts may be first-class artifact references because they are portable
Tree Calculus data checked by pure Tree Calculus code. They are not
Haskell-private semantics.

Source and build provenance are intentionally excluded from the first module
manifest format. A module manifest answers:

```text
What portable artifacts does this module export, and what portable contracts are
paired with them?
```

It does not answer:

```text
Which source file, parser, frontend, or build command produced these artifacts?
```

## 3. Hashing Convention

Objects are content-addressed by SHA-256 over domain-separated canonical bytes.

General rule:

```text
hash = SHA256(domainUtf8 || 0x00 || canonicalPayloadBytes)
```

This matches the existing Merkle node convention in `Research.nodeHash`:

```text
SHA256("arboricx.merkle.node.v1" || 0x00 || nodePayload)
```

The domain string is part of the object format. It prevents identical payload
bytes in different formats from accidentally sharing identity.

Hashes are represented externally as 64 lowercase hexadecimal characters.

## 4. Filesystem Store Layout

The canonical filesystem store layout is:

```text
store/
  objects/
    abc/
      abc123...        -- object bytes, sharded by first 3 hex chars
  aliases/
    names/
    modules/
    packages/
  manifests/
  tmp/
```

The three-character shard follows the existing `lib/arboricx/server.tri`
convention.

### 4.1 Object paths

For object hash:

```text
abc123...
```

object bytes live at:

```text
store/objects/abc/abc123...
```

The object filename is the full hash. The shard directory is the first three hex
characters.

### 4.2 Atomic writes

Writers should use:

```text
store/tmp/<hash>.<nonce>.tmp
```

then atomically rename into:

```text
store/objects/<shard>/<hash>
```

Writing an existing object is idempotent if the existing bytes match the hash.

### 4.3 Store core metadata

The minimal filesystem store does not require sidecar metadata for every object.
Object kind can be known by context or by manifest references.

A later index may cache:

```text
hash -> kind
hash -> size
hash -> references
hash -> createdAt
```

but this index is not semantic identity.

## 5. Arboricx Merkle Node Object Format

The persistent Tree Calculus representation is a Merkle DAG of node objects.

Domain:

```text
arboricx.merkle.node.v1
```

Canonical payloads:

```text
Leaf         = 0x00
Stem child   = 0x01 || childHashRaw32
Fork left right
             = 0x02 || leftHashRaw32 || rightHashRaw32
```

Where `childHashRaw32`, `leftHashRaw32`, and `rightHashRaw32` are the raw 32-byte
SHA-256 digests corresponding to child node hashes.

This is already implemented conceptually by:

```text
Research.Node
Research.serializeNode
Research.deserializeNode
Research.nodeHash
```

The filesystem CAS should use this payload/hash convention directly.

## 6. Tree Roots

A Tree Calculus value stored in the CAS is identified by the hash of its root
Merkle node.

```text
treeRootHash = hash(rootNodePayload)
```

The complete tree is reconstructed by recursively loading node objects reachable
from the root.

Hydration is an interpretation step, not part of object identity. A client may
hydrate a root as a plain tree, a graph with explicit sharing, or another runtime
representation as long as the observable Tree Calculus value is the same. The
filesystem CAS provides structural dedupe and portable identity; it does not by
itself guarantee that a hydrated runtime value is the cheapest representation for
all workloads.

Merkle nodes are useful for explicit DAG-oriented tooling, audit, and bundle
packing. They are not the default representation for module executable exports:
storing every subtree as a separate filesystem object is pathologically slow for
large normal forms.

For module-backed evaluation and imports, a complete normalized named term is
stored as one canonical object:

```text
kind: arboricx.tree-term.v1
hash: <whole-term object hash>
abi:  arboricx.abi.tree.v1
```

The `arboricx.tree-term.v1` payload is a prefix encoding:

```text
Leaf     = 0x00
Stem t   = 0x01 Tree
Fork l r = 0x02 Tree Tree
```

## 7. Arboricx Indexed Bundles

Indexed `.arboricx` bundles remain the transport/execution format.

They are:

- compact;
- self-contained;
- deterministic;
- suitable for restricted runtimes;
- suitable for HTTP serving and deployment.

They are not the canonical long-lived deduplicated store representation.

### 7.1 Pack

Packing converts one or more CAS tree roots into an indexed bundle:

```text
CAS tree roots -> indexed Arboricx bundle
```

The packer traverses reachable Merkle nodes, emits a compact indexed node table,
and writes a bundle manifest with export names and root indices.

### 7.3 Unpack

Unpacking converts a bundle into CAS nodes:

```text
indexed Arboricx bundle -> CAS tree roots
```

The unpacker verifies the bundle structure, reconstructs each exported tree, and
stores the corresponding Merkle nodes. It returns the tree root hash for each
bundle export.

## 8. Module Manifest v1

A module is an immutable manifest object. The module identity is the hash of its
canonical manifest bytes.

A module name is not identity. It is a workspace alias to a module manifest hash.

### 8.1 Domain

Proposed domain:

```text
arboricx.module-manifest.v1
```

### 8.2 Purpose

A module manifest pairs human-facing export names with portable content objects
and optional portable contracts.

It exists to support:

- reproducible import resolution;
- executable export discovery;
- Contract lookup for imported symbols;
- module-to-module reference tracking;
- transport/store interop.

It does not describe source provenance.

### 8.3 Conceptual shape

```text
moduleManifestV1:
  imports:
    - alias: <text>
      kind:  <object kind>
      hash:  <object hash>

  exports:
    - name: <text>
      object:
        kind: <object kind>
        hash: <object hash>
        abi:  <abi identifier>
      contract: optional
        kind: arboricx.tree-term.v1
        hash: <contract term hash>

  metadata: optional human-facing fields
```

### 8.4 Imports/references

The `imports` section is a manifest reference graph, not a store-level language
dependency graph.

Each entry records direct content-addressed references used by the module:

```text
alias: Prelude
kind:  arboricx.module-manifest.v1
hash:  <module hash>
```

This supports reproducibility, partial fetch, and audit. The content store core
stores this object but does not need to understand `Prelude` or import
semantics.

### 8.5 Exports

Each export is a record, not a single hash. This is required so executable
objects and advertised contracts cannot drift apart.

Minimal executable export:

```text
name: "id"
object:
  kind: arboricx.tree-term.v1
  hash: <whole-term hash>
  abi:  arboricx.abi.tree.v1
```

Export with Contract:

```text
name: "map"
object:
  kind: arboricx.tree-term.v1
  hash: <whole-term hash>
  abi:  arboricx.abi.tree.v1
contract:
  kind: arboricx.tree-term.v1
  hash: <contract term hash>
```

The manifest preserves the pairing between exported executable and exported
contract. For workspace modules built from local source, annotated exports are
checked before the manifest is published; only exports that pass producer-side
checking receive direct contract term refs.

### 8.6 Metadata

Metadata is optional and human-facing. Initial fields may include:

```text
package
version
description
license
createdBy
```

Metadata is not source provenance and is not required for execution or checking.

## 9. Contract Artifacts

Contracts are ordinary `tricu` functions `Tree -> Result Tree Tree`. They are
stored and referenced as ordinary `arboricx.tree-term.v1` objects. There is no
separate contract object kind.

A contract object is a complete Tree Calculus term. Any implementation that can
evaluate Tree Calculus terms can apply it. The contract standard defines only the
result convention and the boundary helpers; it does not define a binary contract
grammar.

### 9.1 Export-level pairing

The module manifest pairs each export with an optional contract object:

```text
name: "map"
object:
  kind: arboricx.tree-term.v1
  hash: <whole-term hash>
  abi:  arboricx.abi.tree.v1
contract:
  kind: arboricx.tree-term.v1
  hash: <contract term hash>
```

This prevents the executable and its advertised contract from drifting apart.

### 9.2 Import checking

When a source file imports a contracted export, the frontend loads the contract
object and applies it at the boundary using the standard contract helpers. For
example:

```text
imported List.map has contract <tree-term hash>
```

For locally built workspace modules, advertised export contracts may be checked
before the manifest is published. For external or prebuilt manifests, the
advertised contract is a trusted boundary declaration; the consumer may insert
guard wrappers as needed.

The contract term itself is the authority. There is no separate checker binary
format and no typed-program evidence graph.

### 9.3 Execution hydration versus contract checking

Execution imports use a narrow path:

```text
module import -> selected executable exports -> hydrate selected tree-term objects
```

Contract-aware imports use a slightly broader path:

```text
module import -> selected exports -> exported contract term refs -> apply at boundary
```

Because contract objects are ordinary tree terms, they can be reused, composed,
and stored with the same tools as any other value.

## 10. Workspace Aliases

A workspace is mutable human-facing state over immutable content.

Examples:

```text
List       -> module manifest hash
Prelude    -> module manifest hash
map        -> tree-term hash
httpServer -> bundle hash
```

Aliases should live under:

```text
store/aliases/
```

Initial categories:

```text
store/aliases/modules/<name>
store/aliases/names/<name>
store/aliases/packages/<name>
```

Alias file contents should be simple and explicit, for example:

```text
kind: arboricx.module-manifest.v1
hash: abc123...
```

Exact encoding can be decided with the first implementation. The important rule
is that aliases are mutable pointers, not content identity.

## 11. Existing Convention Alignment

This design intentionally preserves existing conventions where they already fit:

- SHA-256 domain-separated Merkle node hashing;
- `Leaf` / `Stem` / `Fork` node payload tags `0x00`, `0x01`, `0x02`;
- three-character object sharding from `lib/arboricx/server.tri`;
- indexed Arboricx bundles as compact transport objects;
- optional human-facing export names in manifests;
- Contract terms as portable Tree Calculus data.

It replaces or demotes conventions that do not fit:

- SQLite `terms.names` comma-separated aliases become workspace aliases/indexes;
- SQLite `terms.tags` comma-separated tags become optional metadata/indexes;
- file imports as AST flattening become transitional behavior;
- names cease to be semantic identity.

## 12. Implementation Sketch

A staged implementation can proceed as follows:

1. Add filesystem CAS helpers alongside the existing SQLite store.
2. Store/load Arboricx Merkle nodes using the filesystem layout.
3. Implement tree-term storage and reconstruction from filesystem CAS.
4. Implement pack from CAS tree terms/Merkle roots to indexed Arboricx bundle.
5. Implement unpack from indexed Arboricx bundle to CAS tree terms/Merkle roots.
6. Define a concrete module manifest encoding.
7. Store/load module manifests as content-addressed objects.
8. Add workspace alias read/write helpers.
9. Teach import resolution to target module manifests/exports.
10. Attach exported contract terms to module exports.
11. Gradually migrate existing `!import` users.

## 13. Deferred Decisions

These are intentionally left out of the first concrete format:

- package version solving;
- registry/remotes protocol;
- garbage collection/reachability;
- source/provenance/build-record objects;
- editor/update workflows;
- rich visibility/export rules;
- final import syntax;
- whether module manifests also need a tree-native encoding.

## 14. Summary

The concrete v1 direction is:

```text
Store:
  filesystem-backed content-addressed objects

Hashing:
  SHA256(domain || 0x00 || canonical payload)

Tree persistence:
  Arboricx Merkle nodes

Transport:
  indexed .arboricx bundles, packable from and unpackable to CAS roots

Modules:
  immutable manifests pairing export names with object refs and optional
  contract term refs

Workspace:
  mutable aliases from human names to immutable content hashes
```

This keeps the store portable, preserves Arboricx's compact transport role,
restores Merkle DAGs as the persistence model, and gives contracts a stable
module/export attachment point without making the store `tricu`-specific.