--- output_file: "Lattice - Forrest Orchestration.md" feed: "WHITEPAPER" template_version: "1.2" source: "Lattice" codex_section: "S16-S20" conv_id: "lattice-build-history" share_url: "none" date: "2026-07-02" content_date: "2026-07-02" enriched_date: "2026-07-02" date_modified: "2026-07-02" message_count: 1 type: "lattice-build-spec" status: "deposited" path_hint: "IntelligentNetware/Lattice/" category: - "Lattice Infrastructure" - "Forrest Agent" tags: - "lattice" - "forrest" - "orchestration" - "corpus-expedition" - "json-contract" - "brief" monomyth_stage: "07 - Approach to the Inmost Cave" gameboard_position: "Host-side expedition pipeline; model does judgment only" summary: | Forrest corpus expedition decomposition spec: monolithic agent turn fails because retrieval gates and file reads are deterministic work belonging in app.py not the model. Six-step pipeline with three JSON model calls; two-turn gate returns Dig Report before brief assembly. voice_anchor and banned-phrase gates enforced in code.. keypoints: - Design invariant: one bounded JSON transform per model call; orchestration in app.py - Steps: term-gen retrieve score pick_scotoma extract_excerpts assemble_brief - Two-turn gate: handler stops after Dig Report; brief only on confirm - body-not-frontmatter gate rejects digest-layer excerpts for voice_anchor sentiment: archival emotions: - "purposeful" sycophancy_level: "None" truth_score: 95 entropy_score: "Low" codex: true pre_silicon: false ma5_inheritance: "IntelligentNetware" voice_print: false lattice_release: "3.0" original_handoff: "SPEC-Forrest-Decomposition-v1.md" --- # BUILD SPEC — Forrest Corpus Expedition, Orchestrated **Version:** 1.0 · 2026-07-02 **Author:** Claude (Axial Refinement Sherpa), at Daniel J. Comp's direction **Target file:** `app.py` (IN Lattice server, release 3.0) **Status:** draft for Grok (Master Reference Sherpa) review + Carbon Steward ratification --- ## 1. The core reframe The current failure is not a model-capability ceiling and not a charter gap. It is an **architecture mismatch**: Forrest's brief is produced by a single free-form agent turn (`run_conversation` → `AIAgent` with `max_iterations=25`, SOUL-driven) in which the model is asked to run a four-pass dig *and* assemble a brief in one open-ended request. Given that shape, Grok 4.3 reliably shortcuts to the cheap half (emit a brief) and skips the expensive half (read many summaries, stop and wait, open files, print gates). Every one of those skipped behaviors is deterministic work that does **not** belong to the model. The fix is to keep the model for the small bounded judgments it already does reliably (generate terms, score a handful of summaries, pick a scotoma) and move everything else — retrieval, sequencing, file reads, excerpt slicing, gate enforcement — into `app.py`. **Design invariant:** No single model call is ever asked to do more than one bounded transform, and every model call is JSON-in / JSON-out so it is checkable before the next call fires. The dig *emerges from orchestration*; no call requires held-across-turns effort. **Model-agnostic:** because each step is a narrow JSON contract, the underlying model is a config choice. Grok 4.3 should clear every step. If any single step proves shaky, that step (and only that step) can be routed to another model via the existing `mcfg` config without touching the pipeline. --- ## 2. What already exists in app.py (reuse, do not rebuild) Confirmed by reading the source: - **Model invocation:** `_build_bridge_script(...)` + `_run_bridge_script(container, script, timeout)` docker-exec a Python body into the Hermes container, which runs `run_agent.AIAgent(...).run_conversation(msg)` and returns text between `===FINAL===` / `===END===` markers. (Around lines 12700–12775.) - **KB is mounted** read-only at `/opt/kb` inside the `daniel` container; Forrest profile lives there too. `corpus_index.md` at `/opt/kb/corpus_index.md`. Path resolution rule already codified: `'/opt/kb/' + file_path.removeprefix('Corpus/')`. - **Brief handler:** `brief_button_handler` (line 12109). Brief block formatter `_format_forrest_brief_block` (line 9977). Brief files land in `Agent_Briefs` on the Windows host. - **Dachs research loader:** `_load_dachs_research_for_forrest` (line 9641). - **execute_code tool** is available to the Forrest agent inside the loop (SOUL note lines 246–247). **Implication:** we do NOT need a new HTTP client or model plumbing. We need a new orchestration function that calls the existing bridge **multiple times with narrow prompts**, does the KB reads in host-side Python (or via a minimal exec helper), and enforces gates in `app.py` between calls. --- ## 3. Two build options — pick one ### Option A (recommended): host-side orchestration, model does judgment only `app.py` reads `corpus_index.md` and the KB files **directly on the host** (the index is already copyable to the host, or read via a one-shot `docker exec cat`), runs all search/slicing/gates in Python, and calls the bridge only for the 3 judgment steps (term-gen, summary-scoring, scotoma-pick). - **Pros:** retrieval and gates are pure Python — fully deterministic, debuggable, no model can skip them. Fastest to a reliable result. Excerpt-slicing (body-not-frontmatter) becomes a regex, not a request. This is the architecture that actually ends the loop. - **Cons:** `app.py` needs read access to the KB files. If the KB only exists inside the container, add a tiny `_kb_read(path)` helper that does `docker exec daniel cat ` and caches. One helper, ~10 lines. ### Option B: in-container orchestration script Replace Forrest's free-form turn with a **single fixed Python script** bridged into the container that runs the whole pipeline internally (search + slice + gate in the script, model called via the in-script agent object for the 3 judgments) and prints the finished brief + dig report. - **Pros:** all file access stays in-container; no host KB read needed. - **Cons:** harder to debug; the deterministic logic lives in a generated script string rather than first-class `app.py` code; gates are one exec away from the server's control. **Recommendation: Option A.** The whole point is that the un-skippable steps become code the *server* owns. Spec below is written for Option A; Option B notes appear where they differ. --- ## 4. The pipeline — six steps, three model calls ``` STEP 0 build_expedition_context (app.py, pure) → context dict STEP 1 gen_search_terms (MODEL call #1) → 12 terms [validated] STEP 2 retrieve_candidates (app.py, pure) → scored rows STEP 3 score_summaries (MODEL call #2, batched) → relevance scores [validated] STEP 4 pick_scotoma (MODEL call #3) → 1–2 deep cuts + reasons [validated] STEP 5 extract_excerpts (app.py, pure) → verbatim body excerpts [gate: body-not-frontmatter] assemble_dig_report (app.py, pure) → Dig Report → RETURN TO DANIEL, STOP --- two-turn gate: Daniel confirms/drops/redirects --- STEP 6 assemble_brief (app.py, pure) → brief YAML [gates: voice_anchor, banned-phrase] ``` Three model calls total. Everything else is `app.py`. Note Step 5 + Dig Report is where the **two-turn gate** lives: the server returns the report and *stops*. Brief assembly (Step 6) only runs on the confirming request. The model never decides to stop — the handler simply returns. --- ## 5. Step contracts (JSON schemas) All model calls use the existing bridge. Each prompt ends with a hard instruction: `Return ONLY valid JSON matching this schema. No prose, no markdown fences.` Parse with a tolerant loader (strip accidental ```json fences, then `json.loads`); on parse failure, retry once with the parse error appended, then fail loud with a flag — never fabricate. ### STEP 0 — build_expedition_context (pure) Input: the confirmed intake fields + any blaze artifact path. ```python context = { "keyword_primary": str, "title": str, "slug": str, "voice": "Carbon" | "Si-C" | "Silicon", "archetype_primary": str, "monomyth_hint": str | None, # if intake supplied one "blaze_path": str | None, # role:guidance source, if any "blaze_claims": [str], # extracted from blaze for term fan-out seed } ``` No model. Pull blaze claims/open_questions with a YAML/regex read if `blaze_path` set. ### STEP 1 — gen_search_terms (MODEL call #1) Prompt inputs: `keyword_primary`, `title`, `archetype_primary`, `blaze_claims`. Instruction: derive concept-adjacent search terms — synonyms, causes, effects, physical images, opposites — for finding corpus files that *embody* this pattern. **Output schema:** ```json { "terms": ["string", ... 12 items] } ``` **app.py validation (hard, deterministic):** - exactly 10–12 terms; else retry once. - **≥6 terms share NO whole word with** `keyword_primary` + `title` + blaze named phrases (lowercase, split on non-alpha, stopword-filtered). This is the v2.4 "Pass 1 in costume" guard, enforced in code. If <6 distinct, append the failing terms to the prompt and retry once, then proceed with whatever distinct terms exist and flag `[TERM_FANOUT_WEAK]`. ### STEP 2 — retrieve_candidates (pure) No model. For each term, substring-match (case-insensitive) against the **summary and keypoints columns** of every `corpus_index.md` row (not just tags). Union the hits. Also run Pass 3 structural filter: rows whose `category`/archetype and `monomyth` match `context`. Produce candidate rows with a cheap prescore = (# terms matched) + (structural match ? 2 : 0). Cap the candidate set at ~40 rows (highest prescore) to keep Step 3 bounded. ```python candidates = [ { "file": str, "summary": str, "keypoints": str, "tags": str, "monomyth": str, "prescore": int }, ... ] ``` ### STEP 3 — score_summaries (MODEL call #2, BATCHED — 10 rows per call) This is the discrimination task, and the reason batching matters: the model only ever sees 10 fully-visible summaries at once. It is never asked to "read 30 and remember" — `app.py` paginates and concatenates results. Per batch, prompt inputs: the pattern (one sentence from context) + 10 `{id, summary}` pairs. **Output schema:** ```json { "scores": [ { "id": "row_3", "embodies": 0-3, "why": "≤12 words" }, ... ] } ``` `embodies` = how strongly this summary embodies the article's pattern (0 none … 3 exact). **app.py validation:** every input id returns exactly once; scores in {0,1,2,3}; else retry that batch once. Merge all batches; sort by (embodies desc, prescore desc). ### STEP 4 — pick_scotoma (MODEL call #3) Input: the top **8** scored rows only (small, fully visible). Instruction: choose the 1–2 rows that embody the pattern **without naming its keyword** — the non-obvious deep cut — and state the non-obvious connection in one line each. If none qualify, return empty and say so. **Output schema:** ```json { "deep_cuts": [ { "id": "row_7", "connection": "one line" }, ... 0-2 items ], "direct_hits": [ "row_2", "row_5" ], // strongest keyword-obvious rows, for the report "pass4_empty": false } ``` No new judgment beyond this — `app.py` takes it from here. ### STEP 5 — extract_excerpts (pure) + BODY-NOT-FRONTMATTER GATE No model. For each chosen file (deep cuts + any direct hits Daniel will see): 1. `_kb_read(path)` the full file. 2. Split on the first `---\n...\n---` frontmatter fence. **Excerpt is taken only from BELOW the fence** (the body). 3. **Gate — reject digest:** if the sliced excerpt's first 200 chars have >0.8 similarity to the file's own `summary` frontmatter field OR to the `corpus_index` summary cell for that file, it's the digest layer — reject, re-slice deeper, flag `[EXCERPT_DIGEST_REJECTED:file]` if no body prose found. 4. **Gate — third-person tell:** if excerpt matches `^\s*Daniel\s+(recounts|reflects|records|describes)` it is digest narration → reject, re-slice. 5. Excerpt length: 150–500 words, whole paragraphs (don't cut mid-sentence). ### assemble_dig_report (pure) → RETURN + STOP (two-turn gate) `app.py` composes and returns to the UI: ``` === DIG REPORT === Direct hits: [file] — [one-line reason] ... Deep cuts: [file] — [non-obvious connection] ... Voice anchors:[VoiceLog/transcript files with first-person body prose on this ground] ... Searched: [12 terms] · [n] summaries scored across [k] batches === END DIG REPORT === Reply to confirm, drop, or redirect any candidate before I build the brief. ``` Handler returns here. **No brief is assembled on this turn.** The confirming reply triggers Step 6. ### STEP 6 — assemble_brief (pure) + VOICE_ANCHOR + BANNED-PHRASE GATES No model. Build the brief YAML from confirmed candidates. Two hard gates before write: - **voice_anchor gate (Carbon only):** at least one confirmed `role: lived` excerpt must be verbatim first-person prose. Detect first-person by presence of `\bI\b`/`\bmy\b`/`\bwe\b` in ≥2 paragraphs AND absence of the third-person-Daniel tell. Write the frontmatter line: `voice_anchor: {file} — first-person body prose confirmed, {n} paragraphs` If none qualifies: write `voice_anchor: MISSING` and DO NOT save the brief — return the MISSING prompt to Daniel (record a note / point to a file / dictate live). Malformed brief = not saved. This is the schema validator, in code. - **banned-phrase gate:** harvest distinctive phrases from every `role: guidance` excerpt (blaze + Minyan). Store as `banned_phrases` in the brief frontmatter so Arnie's downstream Step 3.5 has the exact list. (Forrest copies guidance verbatim — no paraphrase — so the harvest is stable.) Write brief to `Agent_Briefs`, reusing existing brief-write logic in `brief_button_handler`. --- ## 6. Where each piece lives in app.py | Concern | Location | Nature | |---|---|---| | Orchestrator entry | new `async def forrest_expedition(context)` called from `brief_button_handler` / research handler | new | | Model calls | existing `_build_bridge_script` + `_run_bridge_script`, one narrow prompt each | reuse | | KB read | new `_kb_read(path)` → `docker exec daniel cat` + in-proc cache (Option A) | new, ~10 lines | | Index parse | new `_parse_corpus_index()` → list[dict] keyed by column | new | | Term validation | new `_validate_fanout_terms(terms, context)` | new, pure | | Excerpt slice + gates | new `_extract_body_excerpt(file_text, index_summary)` | new, pure | | Two-turn stop | `brief_button_handler` returns Dig Report; confirm-branch runs Step 6 | edit handler | | voice_anchor + banned-phrase | new `_finalize_brief(candidates, context)` | new, pure | Forrest's SOUL still governs the **judgment** prompts (term sense, scotoma taste) — the pipeline injects the relevant SOUL guidance into each narrow prompt rather than loading the whole charter per call. The parts of SOUL v2.4 that describe *procedure* (four passes, two-turn gate, gates) become code and can be trimmed from the model-facing charter later. --- ## 7. Failure handling — never fabricate - Any model-call JSON parse failure: one retry with error appended, then return `[STEP_n FAILED — {reason}]` to Daniel. Never invent output. - `pass4_empty: true`: legitimate — Dig Report says "Pass 4 empty; pattern may be new to corpus." - voice_anchor MISSING: brief not saved; Daniel prompted. This is correct behavior, not an error. - All flags surface in the Dig Report / handler response, visible to Daniel, never swallowed. --- ## 8. Test plan (the honest one) The machine-shop find in prior runs proves nothing — it was the charter's own worked example. Validate on a **fresh brief whose deep cut is not pre-supplied**: 1. Pick a new title from `SynNA_00_TitleList`. 2. Run the orchestrated pipeline. Confirm: 12 terms with ≥6 distinct (Step 1 gate visible); a Dig Report returns and the handler STOPS (two-turn gate); the deep cut is a file no one named in the intake; the voice anchor is real first-person body prose, not a summary. 3. Confirm → brief assembles with `voice_anchor:` populated and `banned_phrases:` present. 4. Hand that brief to Arnie v2.0. If his CARBON_GATE passes on first or second revision and the prose carries the voice — the loop is closed end to end. If the orchestrated pipeline produces a real dig where the monolithic turn did not, the problem was never the model — it was asking one turn to do ten things. If a specific step still wobbles, route that one step's model via `mcfg` and re-test only that step. --- ## 9. Migration order (smallest reversible steps) 1. Add `_kb_read`, `_parse_corpus_index` — pure, testable alone. 2. Add Step 1 + `_validate_fanout_terms`; log output, don't wire to brief yet. 3. Add Steps 2–4; print scored candidates to a debug endpoint; eyeball against a known topic. 4. Add Step 5 excerpt slicing + gates; unit-test on a VoiceLog (first-person) and a Formation file (third-person digest) — confirm the digest is rejected. 5. Wire the two-turn Dig Report into `brief_button_handler`; keep the old free-form path behind a flag for A/B. 6. Add Step 6 + gates; flip Forrest to the orchestrated path; retire the free-form turn once the fresh-brief test passes. Each step ships independently and is revertible. No big-bang rewrite of a 14.6k-line file.