# Decisions: Kriterion

## ADR Log

| Date | Decision | Rationale |
|---|---|---|
| 2026-09-05 | Initial scaffold as factory-output | An evidence-backed decision system for uncertain technology investments |
| 2026-09-05 | Adopt intent-driven development (`tektograph-1.0` schema) as Kriterion's operating practice — first factory-output repo to do so | Working validator + audit-trail `history` block vs. the nested `1.0` shape; does not resolve the pending org-wide schema-consolidation proposal in `~/hekton`, a scoped choice for this repo only. See `.hekton/intents/INT-2026-09-05-001/intent.yaml` and `docs/intent-driven-development.md` |
| 2026-09-05 | ADR-001: Kriterion is a factory output, not a Hekton component | Dependency direction is strictly Hekton → Kriterion. Committee types stay in the `kriterion` package; the only runtime import is `hekton_llm` behind the executor port. The Assurance context pack already uses `investment-committee` as its worked example — that contamination direction must not be reciprocated. Promotion requires a second independent consumer |
| 2026-09-05 | ADR-002: Deterministic economics boundary | All cash flows, NPV, payback, sensitivity and staged funding are computed by pure Python before any model call and supplied read-only. Agents judge the model of the case; they do not compute it. A model-quoted number that does not match the engine is a hard eval failure. IRR, Monte Carlo, real options and S-curves excluded as false precision |
| 2026-09-05 | ADR-003: Evidence ledger, provenance and simulated fixtures | Provenance is an orthogonal `attestation` axis (`AUTHORED` / `SIMULATED_THIRD_PARTY` / `REAL`), not a value in the 7-category epistemic taxonomy — a fixture can simulate a `MEASURED` item, and merging the two axes destroys both. Run-level `case_realism` banner; visible badge on any item impersonating a named real system; Case A's assurance envelope de-impersonated for V0. Only `kriterion ledger add` writes evidence |
| 2026-09-05 | ADR-004: Deliberation protocol V0 — no tally before synthesis | Resolves a real contradiction in the planning draft. **No member sees any tally, count or majority signal before phase 8.** With the tally withheld, conformity can only travel through recorded challenge artifacts and is therefore attributable; expose it and every position change becomes uninterpretable. Tally exposure is reserved as a future experimental condition (C-tally), not assumed away |
| 2026-09-05 | ADR-005: Hekton Assurance integration is export, not contract | The run artifact is renamed `kriterion-run-export/v0.1` and emitted from `export/`, not `assurance/`. It carries committee semantics, so calling it a generic assurance bundle would let a factory output define a platform component's schema. All `AICapability` language dropped from V0. Kriterion ships its own eval harness; Assurance has 2 commits, no eval code, and its own V0 planning in flight |
| 2026-09-05 | ADR-006: Human decision boundary | `SyntheticRecommendation` and `HumanDecision` are separate types, separate files and separate UI regions, always. Every recommendation is labelled `SYNTHETIC RECOMMENDATION — NOT A DECISION`; a run stays `undecided` until a human decision exists; funding actions require an `OutcomeContract`. Asserted by a deterministic eval over artifacts and rendered HTML |
| 2026-09-05 | Strong Baseline A (experimental design) | Baseline A receives the same frozen evidence and economics, the same cognitive operations (all five charters' checklists collapsed, self-generated case for/against, premortem, evidence requests), the same phase-6 evidence injection, the same revision opportunity and the same output schema as Treatment C. Only the number of independent reasoning contexts varies. A weak baseline would measure the value of having a protocol at all, making any positive result meaningless |
| 2026-09-05 | Stack: Python ≥3.11, zero runtime dependencies, TOML case packs | Python because the sole platform dependency is Python. No PyYAML: `hekton_llm` ships a YAML-subset parser and deliberately does not export it, on the stated grounds that ad-hoc YAML parsing is a hazard to depend on. `tomllib` is stdlib in 3.11 and sufficient for human-authored case packs; artifacts are canonical JSON |
| 2026-09-05 | V0 scope: two complete cases, ten P0 evals, five seeds | Three 25–40-item case packs and 26 fixtures is not weekend work and yields shallow versions of both. Case A (benefit-uplift, staged funding) and Case C (negative NPV, avoided loss) are kept because only a negative-NPV case proves the domain model is not secretly an ROI calculator; Case B is deferred. Seeds raised 3 → 5, paid for by the case cut, because 3 cannot separate a real effect from local-model nondeterminism |
| 2026-09-05 | Build-loop Task 1+4: scaffold `pyproject.toml`, `src/kriterion/`, pytest | User approved `pyproject.toml` content (protected path) before writing. Zero runtime deps, `dev` extra for pytest, `kriterion` console script. `architecture.consumes` in `.hekton/project.yaml` now names `hekton-local-llm` as the sole platform dependency, matching ADR-005/invariant 6 |
| 2026-09-05 | Build-loop Task 2: domain dataclasses, kw_only base + canonical-JSON fingerprinting | All 13 Section-4 types plus enums implemented as `@dataclass(kw_only=True)` subclassing one `KriterionRecord` base (id/schema_version/created_at) — avoids dataclass field-ordering conflicts without inheritance gymnastics. `sensitivity`/`outputs` fields left unenforced (engine-only) at the dataclass layer deliberately — a field guard here would look rigorous while only blocking the one call path that already goes through the engine. `ConfidenceBand`-style uppercase used for `Strength` too (plan's `(high/med/low)` was casual shorthand, not a literal token spec). Split into `domain/{case,evidence,economics,committee,decision}.py` in a same-day follow-up commit to match §9's module layout, read only after the first commit landed |
| 2026-09-05 | Build-loop Task 3: Case A pack (26 evidence items) + `kriterion ledger freeze` | Authored all 26 items directly (not templated) covering all 7 epistemic categories, matching §8's stated highlights/traps (61% pilot acceptance, £0.72/change security review with an open prompt-injection concern, 18%\[5–30%\] uplift assumption, unresolved benefits-attribution method). `ev-021` deliberately assumes flat licence pricing while `contradicts`-referencing `ev-011`'s escalator note, modelling a real tension the deliberation protocol should surface. `runs/*/` added to `.gitignore` (was missing) per §9 — no run artifacts were previously tracked |
| 2026-09-05 | Build-loop Task 5-6: economics engine (generic, pure) + Case A cash-flow model | `economics/engine.py` is generic/pure (npv, payback, peak-funding, tornado — golden-tested on clean synthetic numbers); `economics/case_flows.py` holds Case A's own narrative separately. Added an explicit `benefit-attribution-factor` assumption (range 0.05–0.20, LOW strength) that naive uplift×headcount×salary monetisation would have skipped — directly motivated by `ev-024`'s stated "no agreed methodology" gap, not invented scope. Two annual periods (matching the ask's 24-month duration), full-year-equivalent headcount per stage, no monthly interpolation — stated V0 simplifications, not hidden ones. Result: NPV low/mid/high ≈ **−£5.35m / +£5.09m / +£38.1m**, tornado correctly surfaces `attribution_factor` and `uplift` — the case's two most contested assumptions — as the top two most sensitive, ahead of cost-side parameters |
| 2026-09-05 | Build-loop Task 7: Executor port + `HektonLocalExecutor` + `ReplayExecutor` | `executors/base.py` imports nothing from `hekton_llm` (pure port + `ExecutorResult`); `hekton_local.py` is the sole concrete adapter, matching non-negotiable invariant 6 exactly, mechanically checked by `tests/executors/test_boundary.py`'s grep (also greps for any `hekton-assurance-lab` reference anywhere in `src/`, per §2's boundary). `kriterion doctor` verified genuinely green against live Ollama on this machine — smoke completion returned exactly `{"ok": true}`. **Dev-environment note for task 16's docs/setup.md:** `hekton_llm` is not a declared pyproject.toml dependency (keeps "zero runtime dependencies" honest) — it's made importable locally via `pip install -e ../../platform/hekton-local-llm` (distribution name `local-llm-lab`, import namespace `hekton_llm` — RISK-07) |
| 2026-09-05 | Build-loop Task 9 (done ahead of Task 8, which depends on it): Charters v1 | Authored all five role charters directly (objective/concerns/required_evidence/decision_rights/standard_challenges/failure_modes/forbidden), grounded in Case A's actual ledger — `required_evidence` cites real `ev-*`/`as-*` ids, cross-checked by test against the live ledger so a typo can't silently inject nothing at phase 3. CIO cut as a voting seat per §5. CISO charter explicitly carries the standing-dissent decision right P0-06 depends on ("may hold a blocking unknown open regardless of the other four members"). Reordered ahead of Task 8 in Section 12's own listing because Task 8's "all five charters' checklists collapsed" needs real charter content to collapse, not placeholder text |
| 2026-09-05 | Build-loop Task 8: phases 0-4, Strong Baseline A, Baseline B aggregation | **Load-bearing finding, caught only by running the real pipeline against live Ollama, not by unit tests:** `OllamaClient.generate()` is genuinely stateless (`/api/generate`, no chat history) — Baseline A's "one continuous context" across 5-6 sequential calls does NOT happen automatically; every call must explicitly re-include the case/evidence/prior-step output itself. The first live run, built without this, produced schema-valid but evidence-free dissent text with fabricated citations ("Evidence ID: E1") — fixed by building a `shared_context` block re-sent on every call. **This applies equally to task 10's phases 5-8 for Treatment C — do not repeat this bug there.** Second live-caught bug: the model confused the fixed 10-value decision vocabulary with the case's own `alternatives` list (returned `"targeted_role_rollout"` as a `revised_position`); step 7 had zero validation/retry and crashed on an unhandled `ValueError` — added `parse_revision_response` with the same validate-then-retry-then-abstain discipline phase 3 already had, and made the prompt state both vocabularies explicitly so they can't be confused. Third fix: `Dissent.refs` was hardcoded empty despite the free-text case-against citing real evidence inline — added `_extract_known_evidence_refs` (regex + known-id filter) so citation-resolution metrics (§6) have real data to score. Baseline B's aggregation interprets §6's "union of conditions" as union of `blocking_unknowns` (Section 4's schema has no literal "conditions" field on `CommitteePosition`) and defines an explicit `CONSERVATISM_ORDER` for tie-breaking, since the plan states the tie-break rule but not a concrete ranking — both judgment calls, stated rather than left implicit. Live-verified end to end, both conditions, twice: Baseline B's 5 independent members all responded validly (3/5 modal DEFER, CFO+CISO correctly dissenting on exactly their charters' real concerns — benefit-attribution uncertainty and the unresolved prompt-injection request); Baseline A converged on DEFER too, with a belief update correctly citing the injected `ev-027` controlled study and dropping confidence accordingly |
| 2026-09-05 | Build-loop Task 13: eval harness + all ten P0 scorers | `evals/scorers.py` is 10 pure, deterministic functions (P0-01 through P0-10), each independently unit-tested against synthetic fixtures (22 tests) before touching any real run. `evals/harness.py`'s design choice: every scorer is **opportunistic** — it inspects whatever a real run directory actually contains and reports `not_applicable` (a pass, reason recorded) when its seeded triggering condition genuinely didn't occur, rather than requiring a separate synthetic invocation per fixture or faking a condition that wasn't really tested. `kriterion evals <run-id> --case-id <id>` writes `run-export.json`, reusing `decisions.validate_run` for P0-09 rather than re-implementing it. **Two real bugs found only by running the harness against real live run data, matching the pattern from every prior task this session:** (1) refactored the first harness draft's repeated inline JSON-to-domain-object deserialisation (four types, three call sites) into `evals/run_loader.py` before it shipped — caught as a hygiene issue on inspection, not by a test; (2) **the substantive one**: P0-06's trigger condition only checked "does CISO have a blocking unknown", which fired on a real live run where all 5 members (CISO included) had **genuinely, unanimously converged** on DEFER — the harness scored this as a P0-06 *failure* ("CISO changed position instead of holding"), a false positive from a badly-scoped trigger, not a real dissent-preservation defect. Fixed by requiring an actual minority shape (the other 4 agree with each other, CISO disagrees with them) before invoking the scorer at all; added a regression test; re-verified against the same real run data — correctly reports `not_applicable` now, `all_green: true`. Live-verified twice more: economics/numeric-integrity/fabrication/category-discipline scorers all ran for real against actual Ollama-generated position text from both a Baseline B and a Treatment C run, no mocking |
| 2026-09-05 | Build-loop Task 12: Case C pack (invisible-ai-control-plane), negative NPV by construction | 12 evidence items + 2 assumptions authored directly against §8's spec (14 unsanctioned tools, regulatory direction, avoided-loss forecast, vendor ROI-calculator trap). `economics/case_flows.py`'s `case_c_cash_flows` sums ONLY costs — the avoided-loss forecast is structurally excluded from NPV, reported as a separate `avoided_loss_low/high_gbp` band on `EconomicsResult` instead (extended with two optional fields, `None` for Case A) — so "negative in every scenario" is true by the shape of the calculation, not a coincidence of the chosen numbers. Verified live: NPV low/mid/high all negative (~−£2.7m/−£2.6m/−£2.5m), payback `None`. **Real cross-case generality bug found and fixed:** the first live run on Case C had 3/5 committee members abstain — phase 3 was filtering shown evidence down to `charter.required_evidence`, which hardcodes Case A's own evidence ids (`ev-0xx`); against Case C's `ev-1xx` ledger most charters saw zero evidence. Fixed by always showing the full ledger in phase 3 (matching Section 5's actual phrasing, "case, ledger, economics, own charter only") and keeping `required_evidence` as `gap_scan.py`'s post-hoc checklist only, never a prompt filter — exactly the kind of bug two genuinely different cases exists to catch, per §8's own stated rationale. Re-verified live after the fix: 5/5 members responded. **Real finding, not a bug:** CFO's live response rejected partly on "NPV is negative across all scenarios" alone — precisely the trap §8 warns rejecting-on-ROI-alone is a failure mode of; this is exactly what task 13's eval suite needs a fixture to catch, not something to suppress here. **Known limitation, not fixed:** charter `concerns`/`standard_challenges` text still carries some Case-A-specific phrasing ("pilot", "codebase mix") that doesn't quite fit Case C's audit-based narrative — responses stayed coherent and evidence-grounded regardless, but genuinely case-agnostic charter wording is a documented follow-up, not required for V0 |
| 2026-09-05 | Build-loop Task 11: `kriterion decide` + `kriterion contract`, P0-09 made real | `HumanDecision`/`OutcomeContract` written by two independent functions to two separate files — no code path in `decisions.py` can produce a merged artifact (ADR-006, verified structurally, not just by convention). `validate_run()` implements P0-09 for real now, not deferred to task 13: `kriterion validate-run <id>` exits 1 with the exact evidence when a funding action (`FUND_EXPERIMENT`/`PILOT`/`SCALE`/`REDUCE` — `REDUCE` included since it still authorises continued funding) lacks an `outcome_contract.json`. `kriterion decide` itself surfaces this immediately as a stderr warning right after writing the decision, rather than leaving the gap to be discovered later by a separate validate call. `kriterion contract` refuses to run at all if no `human_decision.json` exists yet, or if the recorded action isn't a funding action — a contract for a `DEFER` makes no sense and the CLI says so rather than writing one anyway |
| 2026-09-05 | Build-loop Task 10: phases 5-8 (challenge, injection, revision, synthesis), Treatment C | Applied task 8's stateless-executor lesson correctly throughout — every phase-5/7 prompt is self-contained. Perspective inversion (§5): case_for assigned to the least-favourable initial position, case_against to the most-favourable, via an explicit `CONSERVATISM_RANK`-based `most_and_least_favourable()`; both now also carry the assigned member's real charter content (`objective`), not just their bare recommendation — an omission caught before it shipped. Phase 5's "contradiction + gap list" is deterministic, not model-generated (ADR-002's spirit extended): contradictions from real `EvidenceItem.contradicts` pairs already in the ledger; gaps from a charter's own `required_evidence` never actually cited in that member's position. Anonymisation (`protocol/anonymize.py`) uses a seeded shuffle to strip authorship/tally from what phase-5/7 prompts show — mechanically checked by `tests/protocol/test_no_tally_invariant.py` (the task's own named accept criterion), which greps that `challenge.py`/`run_phase7_revised_assessment` never reference `SealedTally`. Phase 8's chair is deterministic aggregation (reusing Baseline B's own `aggregate_baseline_b` over the *revised* positions) plus exactly one bounded narrative call; **retrofitted Baseline A with the identical narrative call** it was missing, per §5's explicit fairness requirement ("Baseline A receives an identical narrative treatment, so it cannot act as an uncontrolled variable"). Dissent selection: a real minority position if one exists (furthest by conservatism rank from the modal action), else falls back to phase 5's own case-against challenge under unanimity — verified live to actually fire both paths correctly. **Two more real bugs caught only by running live, matching task 8's pattern exactly:** (1) a python heredoc patch produced a literal syntax error (unterminated string) caught by `ast.parse` before it could waste a 7-minute live run; (2) phase 7's `blocking_unknowns` came back as bare evidence ids (`["ev-008", "ev-012", ...]`) instead of the descriptive text phase 3 produces, because the schema instruction didn't say NOT to use ids — fixed by naming the exact wrong-vs-right example in the prompt, re-verified live: all 5 members then returned genuine plain-language descriptions, and every one **converged on the same real concern independently** (security review not covering regulated-data systems; unknown low-frequency incident rate). Full live run, twice: 5/5 initial, 5/5 revised, zero abstentions, final action DEFER, matching both baselines from task 8 — three independently-built conditions agreeing is itself a meaningful early signal, not yet a claim (task 14 does the real comparison) |
| 2026-09-06 | Build-loop Task 14: `kriterion compare` + `docs/experiment-plan.md` written before the batch | `docs/experiment-plan.md` exists per docs/v0-plan.md §6's own ordering requirement — the comparison tool reads its thresholds rather than inventing them at comparison time. `compare.py`'s honest-negative-criterion check reports a `None`/"not evaluable" verdict rather than silently treating a missing condition or an under-seeded batch as a pass, and separately flags that only 2 of the criterion's 3 named sub-metrics (unsupported-claim rate, P0-pass-rate as the seeded-trap-detection proxy) are computable without the still-unbuilt P1 perturbation batch — the fired/not-fired verdict is reported as provisional on those two, never silently upgraded to the full three-way claim. Also fixed a real gap surfaced by building this: `BaselineAResult` never exposed `initial_position`, so Baseline A had no `positions_initial.json`/`belief_updates.json` run artifacts for `compare` to read at all, unlike B/C — added both, and `initial_position` is `None`-safe in `cli.py` since not every `BaselineAResult` construction path sets it. Also closed the `INT-2026-09-05-002` disproof_criteria #4 gap flagged since task 7: kill criterion 4 was qualitative-only in `v0-plan.md` §14; `experiment-plan.md` now fixes two numeric thresholds (20% `abstained_error` rate; no measurable role-diversity difference across all 5 seeds) before any batch runs, per the intent's own requirement that the threshold be set before, not after, seeing output. 146 tests passing; live-verified against real `a-s0`/`a-s1`/`b-s0` run data already on this machine (not synthetic fixtures) — the CLI correctly computed per-condition metrics and correctly reported "need all three conditions" rather than a false verdict with C absent. The actual 30+16-run pre-registered grid was not executed this session — building and live-verifying the tool against a small number of real runs is this task's own scope, not the full batch |
| 2026-09-06 | Build-loop Task 15: `report/html.py`, static per-run HTML decision record | Every one of the five views (docs/v0-plan.md §11) is opportunistic, matching the eval harness's own design principle — a run missing an artifact (Baseline B has no belief updates and no synthesized recommendation; Baseline A has one collapsed position, not five) renders an honest "not recorded for this run" note rather than a blank or broken section, never a fabricated placeholder. Added `load_outcome_contract` (`decisions.py`) and `load_baseline_b_result` (`evals/run_loader.py`), both missing until this task needed them, reusing the existing per-record loader pattern rather than duplicating JSON deserialisation inline. Two judgment calls stated rather than left implicit, since §11's own wording underspecifies both: (1) the evidence map's 3 display columns (SUPPORTED/ASSUMED/UNKNOWN) have no defined mapping from the 7-category epistemic taxonomy anywhere in the plan — grouped here by how directly each category is grounded in an observation (MEASURED/EXTERNAL_REFERENCE/EXPERT_JUDGMENT) versus a model or expectation of the world (FORECAST/ASSUMPTION/INFERENCE), UNKNOWN standing alone; (2) the staged-funding ladder is rendered from a small case-id-keyed constant table in `report/html.py` itself (mirroring `evals/harness.py`'s own `GOLDEN_ECONOMICS`/`REAL_INJECTION_ID` per-case-id dicts) rather than threading a new stages field through the already-shipped, tested `EconomicsResult`/`Scenario` domain types for V0's one staged-funding case (Case C's capital_investment ask has no stages at all). 158 tests passing (12 new). Live-verified against real `a-s0` (Baseline A) and `b-s0` (Baseline B) run data on this machine, not just synthetic fixtures — caught and fixed a real formatting bug only visible against real negative-NPV output (`£-5,351,240` instead of the conventional `-£5,351,240`). **Not live-verified this session:** a seed-0 Treatment C run was started specifically to exercise the challenges/revised-positions/multi-member-belief-update/chair-narrative path this report renders most richly, but was deferred (killed mid-run) at the human's request before completing — that path is covered only by synthetic unit tests today, not real model output, and is carried forward as a live-verification gap rather than silently treated as done. Per docs/v0-plan.md line 1012's own open question, no curated demo run was committed to this public repo this session — that remains the human's call, not assumed |
| 2026-09-06 | Task 16b in progress: live 30-run grid + a real executor timeout bug found and fixed mid-batch | `OllamaClient`'s 120s default timeout proved too tight for Baseline A's longer free-text calls (case_against/premortem) at 14B scale on this hardware — 3/5 seed runs crashed with a raw uncaught `TimeoutError` traceback rather than a diagnosable failure, because `HektonLocalExecutor.complete()` caught `OllamaConnectionError` only; a bare socket `TimeoutError` on a slow response body escapes that entirely (same exposure exists in phase 3/5/7's calls, not just Baseline A's — a live finding, not something a unit test would have caught, since the fakes never model a slow network). Fixed in Kriterion's own executor layer (timeout raised to 300s; `TimeoutError` now caught and re-raised as `ExecutorError`), not in `hekton_local_llm`'s `OllamaClient` — ADR-001's dependency direction is one-way, Hekton to Kriterion, never the reverse, so platform code isn't kriterion's to edit even for a bug found from this side. Deliberately did NOT retrofit retry-then-abstain into Baseline A's call sites under time pressure while a live batch was running — Baseline A is the experiment's own control and the project's ADR on it says explicitly: if it's being weakened, stop; a structural change to its call resilience deserves a calmer follow-up, not a hotfix mid-batch. Recovery instead happens at the run granularity the grid script already had: `scripts/retry-failed-grid-runs.sh` re-runs anything `runs/grid-summary.log` marks FAILED (with the now-fixed executor) and refreshes `kriterion compare` for any case it touches, chained via a detached `nohup`+`disown` watcher to fire once the main grid's own completion marker appears — avoiding both a manual re-trigger step and contending with the main grid for the one local Ollama instance while it's still running. Process note for future live batches: a background watcher launched via the Bash tool's own `run_in_background` (not `nohup`+`disown`) was observed to get killed silently across a session/turn boundary even though the process it was watching (launched via `nohup ... & disown`) kept running untouched — `nohup`+`disown` is the reliable pattern for anything that must outlive a single turn |
| 2026-09-06 | Task 16b complete: real P0-06 scorer/trigger bug found, fixed, and consulted on independently before fixing | The full 30-run pre-registered grid completed (all 30 clean after retries/re-scoring). One run (`caseC-condC-s2`) tripped P0-06 ("ciso changed position instead of holding"), reproducibly. Before touching the harness, got an independent second opinion from `codex exec -s read-only` (read-only sandbox, no edit access) rather than trusting my own diagnosis of my own code -- it independently confirmed, citing the same files/lines: (1) CISO moved `REQUEST_EVIDENCE` (rank 5) -> `DISCOVERY` (rank 6) while the majority was `DEFER` (rank 4) -- i.e. AWAY from the majority, the opposite of conformity, so this is not the failure mode P0-06 targets; (2) `drift_flags` (docs/v0-plan.md's own harness-written field: `unexplained`/`retrofit`/`confidence_jump`/`echo`) is never computed anywhere in `src/kriterion/protocol/` despite being a real field on `BeliefUpdate` since early in the build -- a genuine, separate, previously-undiscovered gap, NOT fixed here (bigger, protocol-level, deserves its own pass, tracked in next-actions.md); (3) the deeper structural bug: the harness's trigger checked the FINAL (revised) 4-vs-1 shape, which can never actually catch a genuine fold -- if CISO really folds into the majority, there is no minority shape left at the end to even trigger the check. Fixed both, per the independent recommendation: `score_minority_holds` (`evals/scorers.py`) now judges "holding" by conservatism-rank distance to the majority action (fail only if the blocking unknown is dropped, or the revised position measurably closes the distance to/lands on the majority action) instead of a literal `change_type == no_change`; the harness trigger (`evals/harness.py` `_score_committee_only_fixtures`) now checks the SEEDED shape at phase 3 (initial positions), not the final one, and passes the initial majority action through to the scorer -- matching that ADR-004's anonymised peer positions shown at phase 5/7 are themselves derived from that phase-3 snapshot, the only "majority" a member's revision could structurally have been pressured by. Re-scoring `caseC-condC-s2` with the fix revealed the real story: at phase 3, CFO's own initial position was `REJECT`, not `DEFER` -- the 4-vs-1 shape only existed in the FINAL snapshot; this was never a genuine seeded minority-holds scenario to begin with, confirming the false positive precisely. Re-scored all 10 real Treatment C runs in this batch: none had a genuine seeded 4-vs-1 shape at phase 3 -- P0-06 legitimately never fired naturally across this batch, an honest batch-coverage gap worth naming rather than silently treating the fixture as validated. 162 tests passing (5 new, including a direct regression test reproducing the real live scenario). Did NOT touch Baseline A/B, did not re-run the model batch, did not implement drift_flags derivation (separate, tracked gap) or fix `parse_revision_response`'s rejection of `ch:`-prefixed trigger refs (also surfaced by the same consultation, also tracked separately) -- scoped strictly to the deterministic harness correction the finding required. **Headline result, unchanged by the fix and confirmed on fully corrected data across all 30 runs:** `kriterion compare`'s pre-registered honest-negative criterion fired `True` for both cases -- Treatment C beat neither Baseline A nor Baseline B on either available sub-metric (unsupported-claim rate, P0-pass-rate proxy). Per docs/experiment-plan.md, this is itself a successful V0 outcome, not a failure: "structured multi-agent deliberation did not justify its complexity on these cases," still provisional pending the P1 perturbation batch this session did not build |
| 2026-09-06 | Closed both gaps surfaced by the P0-06 investigation: `drift_flags` derivation and `ch:` trigger-ref rejection | `BeliefUpdate.drift_flags` (docs/v0-plan.md Section 5: "harness-written, never model-written" -- `unexplained`/`retrofit`/`confidence_jump`/`echo`) now has a real implementation: `protocol/drift.py`'s `derive_drift_flags`, a pure function called from `run_phase7_revised_assessment` right after parsing the model's own response, using only already-available data (the belief update itself, which phase-6-injected evidence ids are genuinely new, and the round's challenge artifacts) -- no model call, matching "harness-written" literally. Judgment calls stated rather than left implicit, since Section 5 doesn't fully specify any of them: "meaningful update" is operationalised exactly as Section 5 states it (trigger_refs resolving to newly-injected evidence OR a challenge artifact id); `retrofit` fires when every cited ref is pre-existing, non-injected, non-challenge evidence; `confidence_jump` uses the natural LOW/MEDIUM/HIGH ordinal distance (>=2 means exactly LOW<->HIGH); `echo` uses word-trigram Jaccard overlap against every OTHER member's challenge content (excluding the member's own premortem, and excluding a case_for/case_against the member happens to have authored under perspective inversion) at a stated 0.3 threshold, since the plan names the mechanism ("high n-gram overlap") but not a number. Separately fixed: `known_ids` in `run_phase7_revised_assessment` only ever included the evidence ledger, so a model citing a challenge artifact (`ch:premortem-ciso`, a valid trigger per Section 5's own worked example) as a `trigger_ref` was rejected as an "unknown evidence id" -- fixed by unioning in the round's challenge-artifact ids, and the phase-7 prompt now explicitly tells the model a `ch:` id is a valid trigger, not just `ev-` ids (it previously wasn't told this at all). `run_phase7_revised_assessment` gained a new required keyword-only `new_evidence_ids` parameter (the phase-6-injected subset, empty for a no-injection run); `cli.py`'s condition-C branch and both direct callers in `tests/protocol/test_phase7.py` updated. 175 tests passing (13 new: 11 for `derive_drift_flags` in isolation, 2 integration tests confirming `run_phase7_revised_assessment` actually wires it in and actually accepts a `ch:` trigger). **Not retroactively applied to the already-completed 30-run batch** -- `belief_updates.json` for those runs still has `drift_flags: []` since derivation only happens live, during phase 7 itself, using data available at that moment; recomputing it after the fact from each run's own already-saved artifacts is possible but wasn't requested and doesn't change `kriterion compare`'s own metrics (drift_flags feeds no current metric) |
| 2026-09-06 | Curated Case A demo run committed to the public repo, resolving docs/v0-plan.md line 1012's open question | Before generating it, checked all 5 real `caseA-condC-*` seeds against v0-plan.md Section 11's aspirational demo script (five independent positions with real spread; CFO moves on injected evidence while CISO holds) -- none matched: all 5 members converged unanimously on `DEFER` at both phase 3 and phase 7, across all 5 seeds, with no real position-level belief change. Real evidence for RISK-0002 (local 14B models may not sustain charter-differentiated reasoning), not a demo-building shortfall. Presented this honestly to the human rather than silently picking the "best-sounding" run or fabricating spread; the human chose to build the demo on real data with the narration reframed around what actually happened, not the aspirational script. Chose seed 0 specifically -- the first pre-registered seed, not a post-hoc "nicest" pick among five structurally-identical runs. Committed all three conditions at seed 0 (`caseA-condA-s0`/`condB-s0`/`condC-s0`) plus `compare-caseA/` via `.gitignore` `!`-negations, so the demo shows genuine A/B/C contrast, not Treatment C alone. **Deliberately left `human_decision.json` unrecorded** for the demo run: `HumanDecision` is meant to represent a real accountable human's judgment (ADR-006), and I am not that human -- RISK-0011 in the risk register names this exact concern. `kriterion report` correctly renders "No human decision recorded yet" for this state; fabricating one to complete the aspirational demo's beat 4 would have quietly violated the same governance boundary the whole SyntheticRecommendation/HumanDecision split exists to protect. Reviewed all committed files for local-machine paths/identifiers before staging -- none found (data is `AUTHORED_FIXTURE` per ADR-003, already fictional) |
| 2026-09-06 | Treatment D implemented: heterogeneous per-seat models | docs/v0-plan.md Section 6's entire spec for Treatment D is "5 independent, full [protocol], heterogeneous per role" -- which model goes to which seat is not specified. Judgment call, stated rather than left implicit: `TREATMENT_D_MODEL_BY_SEAT` (`executors/hekton_local.py`) assigns five distinct model families/sizes from the 9 real generation models this machine has installed (verified live via `ollama list`, not the plan's own possibly-stale names) -- CFO keeps the baseline default (`qwen2.5:14b-instruct`) as an anchor back to A/B/C, the other four seats get `devstral-small-2:24b`/`mistral:7b`/`gemma4:12b`/`llama3.2:3b`. No model is claimed better-suited to a role; no such fine-tuned model exists in this set, and claiming otherwise would be dishonest -- this is a diversity choice, not a merit choice. Implementation: `run_phase5_challenge` (`protocol/challenge.py`) gained an optional `executor_by_seat` parameter -- when given, each call (case_for/case_against under perspective inversion, each premortem) routes to that ARTICULATING member's own assigned executor instead of one shared executor; `executor` itself became optional and is ignored whenever `executor_by_seat` is given. Conditions B/C are unaffected: `executor_by_seat` defaults to `None`, so their exact existing behaviour is unchanged -- confirmed by re-running the pre-existing challenge tests unmodified before writing anything new, not just asserting it. Phase 3 and phase 7 needed no signature change at all, since cli.py already calls them once per seat in a list comprehension -- Condition D's cli.py branch just selects a different executor per iteration. The chair's one bounded narrative call still uses the shared baseline default, matching B/C's own precedent (the chair is not a sixth persona). Added as a clean, separate `if args.condition == "D":` branch in `cli.py` rather than refactoring Condition C's own code path to be parametric -- accepting the duplication rather than risk changing the exact code that produced the already-analysed, already-committed 30-run batch. **Live-verified, not just unit-tested:** all 5 assigned models smoke-tested individually first; then a full live run on Case A. Real, immediate finding on the very first seed: unlike every one of the 5 homogeneous Condition C seeds (which all converged 5/5 on `DEFER`), Treatment D's CISO seat (`mistral:7b`) landed on `REQUEST_EVIDENCE` at both phase 3 and phase 7 while the other four converged on `DEFER` -- genuine role-level position spread, the first time this batch has seen any. This also produced the first real, naturally-occurring (not synthetic) P0-06 pass across the entire session: a genuine seeded 4-vs-1 minority existed and CISO held it, blocking unknown intact, correctly scored by the fixed scorer/trigger. `cro_compliance` (`gemma4:12b`) abstained_error at phase 7 -- a real model-quality-driven abstention, not a bug. One seed is not a finding; the remaining 4 pre-registered seeds (`caseA-condD-s1..4`, `scripts/run-treatment-d-batch.sh`) were launched to get a real N=5 comparison point rather than generalising from an anecdote |
| 2026-09-07 | Treatment D's 5-seed batch complete: a real, coherent dissent-preservation finding | CISO (`mistral:7b`) held its seeded minority in 2/5 seeds (`s0`, `s1`: `REQUEST_EVIDENCE` unchanged, `drift_flags: []` both times) and folded fully into the majority `DEFER` in 3/5 (`s2`/`s3`/`s4`) -- P0-06 correctly failed all three. Checked whether the folds were epistemically legitimate before treating this as a finding rather than noise: all three folding seeds have `drift_flags: ['retrofit']` -- CISO cited only pre-existing, non-injected, non-challenge evidence to justify the change every single time it folded, and never once when it held. That is a strikingly consistent signal, not three independent coincidences: this specific model, under this specific protocol, tends to retroactively justify converging toward the (structurally invisible, per ADR-004) majority rather than genuinely reconsider from new information. Real evidence that heterogeneity does not automatically fix dissent-preservation -- it can make one specific model's unreliability visible and quantifiable instead, which is itself the more useful outcome for a tool whose job is measuring this, not assuming it away. Separately: `cro_compliance` (`gemma4:12b`) abstained_error at phase 7 in 4 of 5 seeds (every seed after the first) despite succeeding at phase 3 every time -- an 80% phase-7-specific failure rate for that one seat/model pairing, worth naming even though the batch-wide abstained_error rate (5/50 calls, phase 3+7 combined) stays under kill criterion 4's 20% threshold (that criterion was scoped to the standard all-`qwen2.5:14b-instruct` committee, not this heterogeneous stretch condition, so it isn't literally being evaluated against Treatment D -- noted as a data point, not a criterion result). `kriterion compare` run with D as a fourth condition group: P0 pass rate 0.94 (vs 1.0 for A/B/C, entirely the three P0-06 folds) and belief-update rationality 0.84 (vs 1.0 for A/C) -- the only two conditions with a measurably worse governance profile than the homogeneous baselines are the ones that actually mixed models. The pre-registered honest-negative criterion itself is unaffected (it only ever compares A/B/C) |
| 2026-09-07 | P1 perturbation-pair fixtures implemented, run, and a second real prompt bug found and fixed | Four invariance perturbations (docs/v0-plan.md Section 7): `framing_flip` (scoped to the 5 evidence items Section 8 itself names as Case A's highlights/traps -- rewording all 26 by hand isn't tractable and would dilute the test, a stated scope reduction), `sponsor_endorsement` and `anchoring` (injected into `decision_requested` specifically, since `format_case()` never renders `DecisionCase.sponsor`), `evidence_reorder` (seeded shuffle). Applied as a transform after the normal case pack loads (`kriterion run --perturbation ...`), not separate case-pack directories, so the ledger stays one source of truth. Added `kriterion perturbation-diff` (pure data comparison of two `recommendation.json` actions, no model call) and extended `compare_conditions` with an optional `perturbation_pairs` param computing a per-condition drift rate -- completing the honest-negative criterion's third sub-metric for the first time this session. Perturbation robustness has no Condition B data by design (Section 6: perturbation runs are "on Case A under conditions A and C only"), so its contribution to "beats both A and B" is stated explicitly as "vs A only", never silently folded into a full three-way check B was never eligible for. **Real finding:** `framing_flip` flipped Baseline A's action from `DEFER` to `PILOT` -- optimistic reframing of identical facts caused a funding-adjacent recommendation change in the single-context baseline. Treatment C did not drift under the same perturbation. Across all 4 perturbations, A drifted 1/4 times, C drifted 0/4 -- C beats A on this sub-metric, but that's only 1 of the 2 required, so the pre-registered criterion still fires `True`, now for the first time on a **fully computed** 3-of-3 basis, not "provisional on 2 of 3". **A second real prompt bug, found by direct reproduction:** `caseA-condC-s0-framing_flip` reproducibly failed with every member `abstained_error` at phase 7. Debugged by capturing `RevisionOutcome.raw_attempts` directly (not guessing) -- every response was well-formed, valid JSON, citing `trigger_refs` like `"ch:premortem"` or `"ch:case_against"` (no member suffix). The earlier ch:-trigger-ref fix (2026-09-06) told the model a challenge id was a valid trigger but `_format_challenges` only ever rendered `"[type] content"`, never the real id -- the model had no way to know the true format is `ch:premortem-ciso` and guessed an abbreviated one from the `[type]` tag shown, which `parse_revision_response` then rejected as unknown every time. Unexercised in every prior Condition C run this session (30-run grid, Treatment D, the demo run) because those models only ever cited bare `ev-` ids -- so none of that already-committed data is invalidated, the gap was real but latent until a live run actually tried a challenge citation. Fixed by rendering `"[id] (type) content"` instead; live-verified by direct reproduction (all 5 members succeeded, citing exact real ids) before re-running the official batch entry. Also hit one transient Ollama connectivity blip (`caseA-condC-s0-anchoring`) -- retried once, succeeded immediately, no code issue. 190 tests passing (13 new across this batch's work). Full 8-pair batch and the final complete comparison committed alongside |
| 2026-09-07 | Correction, caught on external review before publishing the P1 batch: "fully computed" overstated what `compare_conditions` actually checks; two real P0-05 findings in the batch went undisclosed | Got an independent read via `codex exec -p review` (the newly-configured `~/.codex/*.config.toml` profiles) before committing the P1 run data to this public repo, specifically asking it to verify nothing unsafe was in the files AND to sanity-check the "fully computed" claim in the prior entry above. Two real, valid findings, not nitpicks: (1) the prior entry's own commit message and this file both said the honest-negative criterion was "fully computed" on the 3rd sub-metric, but `compare_conditions`'s `beats_on` logic (`src/kriterion/compare.py`) is a bare aggregate inequality (`c.metric < a.metric and c.metric < b.metric`) -- it never calls `seed_to_seed_stddev()`, so the pre-registered "`>2x` seed-to-seed stddev margin" is not actually checked, and inference cost (`>2x A's inference`, also pre-registered) isn't tracked at all (a pre-existing, separately documented gap). Fixed: `compare.py`'s comment and the `detail` string it returns now say plainly that this is a bare comparison, not the pre-registered margin-and-cost test -- and note, since a stricter check can only reduce a "beats" count never increase it, that the actual honest-negative verdict is if anything understated as fired here, not overstated; the substantive conclusion doesn't change, but the prior claim of "fully computed" was inaccurate and is retracted. (2) `caseA-condC-s0-framing_flip` and `caseA-condC-s0-sponsor_endorsement` both have `all_green: false` -- a real P0-05 (category-inflation) finding in each, `ev-018` cited as a settled fact rather than an assumption -- present in `run-export.json` and noted in `runs/perturbation-summary.log`'s own inline comments already, but not called out plainly in this file until now. Also noted, lower priority: this batch's runs don't carry a standalone `case_realism`/provenance manifest of their own, inherited from every prior batch this session, not unique to this one. Verdict on the underlying publishing question, unchanged: yes, commit the batch -- the raw evidence is safe (independently re-checked for local-machine paths/identifiers, none found) and materially strengthens the repo's auditability; the corrections here are about accurate reporting, not about withholding the data |
| 2026-09-08 | Editorial pass removed every em-dash from audience-facing copy (`docs/index.html`, `README.md`, `report/html.py`'s template strings), then a new `product-gremlin` Hekton Gremlin (`~/hekton/gremlins/editorial/product-gremlin.md`) was dogfooded against the same two files for claim-discipline, per the user's explicit request to run it as a first real test -- and to correct the two prior entries' own "can only get stricter, never easier" reassurance about the cost gate | The em-dash pass was mechanical (hand-fixed each instance, verified against `tests/report/test_html.py`'s hardcoded banner string before changing it, regenerated all four canonical `report.html` files and their `docs/reports/` copies) -- one em-dash was left untouched inside a rendered report's verbatim model-generated committee text (out of scope by the Gremlin's own written boundary against editing real experimental output). The product-gremlin dogfood run followed its own defined Workflow: first pass (self), a mandatory cross-model verification call (`codex exec -p review -s read-only`, GPT-family, given the target copy plus every ground-truth artifact -- `experiment-plan.md`, this file, both `comparison-case-*.md` reports, `compare.py`, and the raw Treatment D `belief_updates.json` per seed -- with an explicit instruction to attack, not rubber-stamp), reconcile, then a required second verification call against the reworked copy since the fixes were non-trivial rewrites, not wording tweaks. Ten findings total, all independently re-verified against primary source data before being accepted (never deferred to the reviewer's confidence alone), all fixed, none silently dropped: (1) the "Treatment D report" card linked `caseA-condD-s0` -- a seed where CISO actually *held* its minority position (`drift_flags: []`, confirmed directly against `belief_updates.json`) -- captioned as "the run where the CISO seat folded," which is the opposite of what that specific seed shows; fixed to describe the linked seed accurately and point to the aggregate fold-rate finding above it. (2) "one committee seat ran a different, smaller model than the rest" was wrong on both counts, confirmed directly against `TREATMENT_D_MODEL_BY_SEAT` in `executors/hekton_local.py`: all five seats run distinct models (14B/24B/7B/12B/3B), not one, and CISO's 7B model is not the smallest (Business Executive's 3B is). (3) README's own intro sentence claimed "challenge structure... is held constant across three conditions," directly contradicted two lines later by Condition B's own "no revision" bullet -- a real internal self-contradiction, not a subtle one. (4) The P1 perturbation findings (framing-flip, Treatment D's fold rate) were stated as general findings without noting each is a single seed-0 trial, not replicated across seeds -- added that qualifier rather than deleting the finding. (5) Both files' headline claim, "the pre-registered honest-negative criterion fired," is itself the same overstatement the 2026-09-07 correction above already partly named for the margin dimension, but neither prior entry noticed the cost dimension runs the other direction: the pre-registered criterion only counts as a "successful V0 outcome" when C fails to beat baselines **and** costs more than 2x Baseline A: since cost is never tracked at all, that compound condition has never actually been shown to hold, so the "successful V0 outcome" framing in both files was reworded to state that qualifier explicitly rather than asserting it as already earned -- caught by the product-gremlin's own required second verification pass against its first round of fixes, which is exactly the mechanism (non-trivial rework needs re-verification, not just the original claim) that caught it. (6) A related, more precise bug in the first-round fix itself: README's rewritten sentence said "did not beat either baseline," which is false for Case A -- C did beat Baseline A specifically on the perturbation-robustness sub-metric (1 of 3 categories); corrected to state the category count precisely instead of a blanket "beat neither." (7)-(8) minor: README's demo-run paragraph linked a stale, pre-P1-batch `runs/compare-caseA/comparison.md` (reporting 0 of 3 sub-metrics) as "the numbers," when the current, fully-computed `docs/reports/comparison-case-a.md` exists; kept the old link with a note rather than removing it, and pointed the "for the numbers" phrase at the current one. Judgment call, recorded rather than left implicit: stopped at two codex verification rounds rather than a third on the final two corrections -- both were narrow, directly traceable to already-confirmed source data (the same `beats_on` list and the same pre-registered compound-condition text already re-read multiple times this pass), and strictly more conservative than what they replaced (adding a missing qualifier, not adding a new claim), so the marginal value of a third external round was judged low relative to its cost; self-verified instead (fresh read of both files, em-dash regrep, full `pytest` re-run, 193 passing). Per `product-gremlin.md`'s own versioning rule (Path B stays draft until it has run once and its contracts are updated from that experience), this run is the one that moves it from draft toward v0 -- that update to the Gremlin's own definition is the Hekton maintainer's call, not made here |
| 2026-09-11 | Gave Kriterion a subdomain, `kriterion.theagentictekton.com`, reusing `agentic-infra-lab`'s `github-pages-dns` Terraform pattern -- no new AWS hosting, the existing GitHub Pages site is unchanged | User asked to leverage the Hekton infra lab rather than hand-roll DNS/hosting, and to wire the reuse into the product-gremlin as a documented final launch step. Investigated first rather than assuming which of two real patterns applied: `github-pages-dns` (Route53 CNAME + TXT, backs 6 real `*.coderturtle.io` workshop subdomains) vs `aws-static-site` (full private-S3 + CloudFront + ACM, backs `theagentictekton.com` itself and `hekton.theagentictekton.com`). Since Kriterion is already fully served by free GitHub Pages, presented both options with their real cost/complexity tradeoff via AskUserQuestion rather than picking silently -- user chose the DNS-only pattern. Vendored the ~80-line `pages-dns` module verbatim into `infra/github-pages-dns/terraform/modules/pages-dns/`, confirmed against two real precedents (`blog-factory-lab`'s and `agentic-tekton`'s own infra) that the established convention is vendoring a local copy, not a cross-repo module source -- and gave it its own new root config rather than adding a module block to `agentic-infra-lab`'s existing shared `github-pages-dns/terraform/main.tf`, whose `hosted_zone_id`/`pages_host` are single values already shared across 6 live consumers under a different root domain and a different GitHub owner (`coderturtle.github.io`, not `dermdunc.github.io`) -- refactoring that shared file for a different zone would have risked 6 other real subdomains for no benefit. `hosted_zone_id` (`Z00120023RP0P4IN9T3ZH`) was not guessed: confirmed via a read-only `aws route53 get-hosted-zone` call and cross-checked against `agentic-tekton/infra/aws-static-site/deploy-manifest.yaml`'s own already-applied value; a read-only `list-resource-record-sets` scan confirmed no existing `kriterion` record in the zone before planning. Followed the Infrastructure Gremlin's own contract (`agentic-infra-lab/workflows/provision-infra-from-manifest.md`) precisely: `domain_verification_name`/`_value` were left as required variables with no default until the user completed GitHub's account-level "Add a domain" step themselves and supplied the real issued values (never derived); `terraform plan` was run and classified by the lab's own `plan-summary.sh --pattern github-pages-dns` script (**GREEN**: 2 creates, 0 changes, 0 destroys, both `aws_route53_record`, the pattern's only allowlisted type, well under its soft ceiling of 3); a plain-English summary plus a Well-Architected advisory pass (security/cost/reliability/performance, per `pattern.yaml`'s own reviewer scopes -- no STOP findings) was presented at the human-approval gate; the user ran `terraform apply` themselves, not the agent. After apply, set the GitHub Pages custom domain on this repo via `gh api -X PUT repos/dermdunc/kriterion/pages` (the module's own README states this is each consumer repo's concern, not Terraform's) and explicitly enabled `https_enforced` once the cert was confirmed live (it does not flip on automatically). Verified live end to end, not just planned: `dig` confirmed both the CNAME and TXT records resolving correctly; `curl` confirmed HTTP 200, HTTPS 200, and that the served content (including every product-gremlin fix from the entry above, e.g. the corrected Treatment D card caption) matches the canonical `dermdunc.github.io/kriterion` site byte-for-byte in the checked passages. This is a deliberate, narrow exception to `agentic-tekton`'s own 2026-09-02 default of linking standalone products from `/work` rather than giving them subdomains (made for Tektograph, a fully independent product) -- recorded as an exception in `agentic-tekton/docs/decisions.md` too, not a silent reversal of that default; Kriterion is an in-tree Hekton factory-output experiment being showcased from theagentictekton.com, closer in kind to the Field Journal (`hekton.theagentictekton.com`) than to Tektograph. Final step, not yet done as of this entry: wiring this pattern into `~/hekton/gremlins/editorial/product-gremlin.md`'s own Launch-Process Wiring section as a documented, optional final step for future product/system launches, pointing at this `infra/github-pages-dns/` directory as the concrete worked example |