GGUF backend (llama)

SEMIF_BACKEND=llama talks to any stock llama-server hosting a GGUF instead of loading torch: the same logit readout over the server's candidate probabilities, plus /plan and runtime model selection. No fork, no local tokenizer or weights; torch stays selectable and remains in the Nix closure.

Run

nix develop
SEMIF_BACKEND=llama .venv/bin/python -m uvicorn semif_api.app:app --host 127.0.0.1 --port 8322

/decide and /decide-batch keep their request shapes; batches run sequentially with server prefix caching and do not claim torch's shared-prefill semantics. /healthz is liveness only — it never loads or warms the remote model (backend_status: not_checked).

Configuration

Environment	NixOS option (services.semif-api)	Default
SEMIF_BACKEND	backend	torch
SEMIF_LLAMA_URL	llamaUrl	http://127.0.0.1:8080
SEMIF_LLAMA_MODEL	llamaModel	Qwen3.5-4B
SEMIF_LLAMA_TIMEOUT	llamaTimeout	600 s per HTTP request
SEMIF_LLAMA_N_PROBS	llamaNProbs	1024
SEMIF_LLAMA_MAX_N_PROBS	llamaMaxNProbs	16384
SEMIF_LLAMA_CACHE_PROMPT	llamaCachePrompt	true
SEMIF_MAX_TOKENS	maxTokens	4096

SEMIF_LLAMA_MODEL is an alias from the server's /v1/models list; the server needs enough context for the prompt plus one token. SEMIF_MODEL/SEMIF_REVISION apply to torch only.

Behavior enforced in code and covered by tests/test_llama*.py: option-letter token checks are cached per model, prompt boundary checks run per decision, missing option scores retry up a n_probs ladder (1024 → 4096 → 16384, then 502 — never invented scores), invalid input is 422.

Response fields

Beyond the torch shape, rows carry choice, model.backend, and a llama metadata block: timings, attempts, n_probs, tokens_cached, cache_n, slot_id. cache_n is the reused prompt count from server timings — note some server builds also report newly cached tokens in tokens_cached. option_logits are full-vocabulary log probabilities (logits up to an additive constant); probabilities remain uncalibrated conditional option scores. forward_seconds covers the completion HTTP call and retries; total_seconds adds prompt preparation.

Standalone probe

.venv/bin/python -m semif_api.llama --timeout 600
.venv/bin/python -m semif_api.llama --input examples/examples-shared.jsonl > results-llama.jsonl

Input is JSONL with id, state, question, and 2–16 options (id + description); omit --input for a built-in example, --input - reads stdin. --url, --model, --n-probs, --max-n-probs, --max-tokens, and --no-cache override. A failed row exits nonzero, leaving partial output.

Planning

/plan runs one /v1/chat/completions with thinking enabled under the universal PLAN_SYSTEM prompt (in semif_api/llama.py); no max_tokens is sent, since it would cap the reasoning trace. See games.md for the learning loop and ../semif-api/README.md for the request/response shape.

Validation

.venv/bin/python -m unittest discover -s tests -p 'test_llama*.py' -v

tests/test_api.py and scripts/validate.sh are torch parity tools, not llama tests. Server API reference: llama.cpp server.