draft-manuscript-galaxy

run:
  pipeline: paper-to-galaxy
  run_slug: draft-manuscript-galaxy
  status: complete

phases:
  - phase: 1
    kind: mold
    skill: summarize-paper
    status: done
    feedback_checked: true
  - phase: 2
    kind: mold
    skill: freeform-summary-to-galaxy-interface
    status: done
    feedback_checked: true
  - phase: 3
    kind: mold
    skill: freeform-summary-to-galaxy-data-flow
    status: done
    feedback_checked: true
  - phase: 4
    kind: mold
    skill: compare-against-iwc-exemplar
    status: done
    feedback_checked: true
  - phase: 5
    kind: mold
    skill: freeform-summary-to-galaxy-template
    status: done
    feedback_checked: true
  - phase: 6
    kind: mold
    skill: advance-galaxy-draft-step
    loop: true
    iterations: 9
    status: done
    feedback_checked: true
  - phase: 7
    kind: branch
    pattern: test-data-resolution
    chain: [paper-to-test-data, find-test-data, user-supplied]
    selected: find-test-data
    status: done
    feedback_checked: true
  - phase: 8
    kind: mold
    skill: freeform-summary-to-galaxy-test-plan
    status: done
    feedback_checked: true
  - phase: 9
    kind: mold
    skill: implement-galaxy-workflow-test
    status: done
    feedback_checked: true
  - phase: 10
    kind: mold
    skill: validate-galaxy-workflow
    status: done
    feedback_checked: true
  - phase: 11
    kind: mold
    skill: run-workflow-test
    status: done
    feedback_checked: true
  - phase: 12
    kind: mold
    skill: debug-galaxy-workflow-output
    status: done
    feedback_checked: true

entries:
  - id: collection-pattern-mocs-ship-without-referenced-pages
    raised_by: freeform-summary-to-galaxy-template
    observed_in:
      mold:
        name: freeform-summary-to-galaxy-template
        path: content/molds/freeform-summary-to-galaxy-template/index.md
        revision: 6
        content_hash: 0a95f8c04465d48131ace523e0b27af53f1ade6e525a9173c5ff1e936ddc3366
        foundry_head: 63a3f9cf9c97a637fe1628d298e524f35709e289
    subject:
      kind: pattern
      label: galaxy-collection-patterns (and sibling galaxy-conditionals-patterns / galaxy-tabular-patterns MOCs)
      locator: content/patterns/galaxy-collection-patterns.md
      content_hash: 9840d686228a254766611196f2be03b2a66119281322c6373ce082cd08e78717
    kind: gap
    severity: minor
    what: "The skill bundle packages `galaxy-collection-patterns.md`, `galaxy-conditionals-patterns.md`, and `galaxy-tabular-patterns.md` as MOC/index pages (frontmatter `pattern_kind: moc`) that only list wikilink-style names of ~15-20 concrete pattern pages each (e.g. `[[fan-in-bundle-consume-and-flatten]]`, `[[collection-unbox-singleton]]`, `[[manifest-to-mapped-collection-lifecycle]]`, `[[tabular-concatenate-collection-to-table]]`). None of the referenced pattern pages themselves -- which per the MOC descriptions should carry the actual worked recipe, concrete tool_id, and state -- are packaged anywhere in this skill bundle's `references/` tree."
    expected: "Either package the linked pattern pages alongside their MOC (as this skill already does for other reference kinds, e.g. `galaxy-workflow-draft-format.md`), or have the MOC entries themselves carry the concrete tool_id / state worked example inline, so a cast skill that is explicitly told (Runtime Notes) not to fetch Foundry source files at runtime can actually resolve a pattern name to a usable recipe from packaged content alone."
    evidence: "For this run, resolving concrete tool identities for the fan-in/flatten/tabular-bridge steps (Collapse Collection, __FLATTEN__, collection_column_join) was only possible because the separately-supplied `iwc-comparison-notes.md` artifact (from a prior pipeline phase) happened to quote inline gxformat2 excerpts naming those tools. Had that artifact not carried those excerpts, the packaged collection/tabular pattern MOCs alone (the on-demand references this Mold's own SKILL.md directs it to for exactly this situation) would have given only a pattern *name* with no way to resolve it to a tool_id or worked state, forcing every such step to Deferred tier even where a concrete built-in exists."
    status: filed
    issue: https://github.com/galaxyproject/foundry/issues/570

  - id: gxformat2-schema-missing-nested-in-key-convention
    raised_by: advance-galaxy-draft-step
    observed_in:
      mold:
        name: advance-galaxy-draft-step
        path: content/molds/advance-galaxy-draft-step/index.md
        revision: 4
        content_hash: c92d452f69a7b40bcab5fc8e63b03e1e2abf98f2a636abc4c93ad7f3c1d03082
        foundry_head: 63a3f9cf9c97a637fe1628d298e524f35709e289
    subject:
      kind: research
      label: gxformat2-schema
      locator: content/research/gxformat2-schema/index.md
      content_hash: 4fe42382f99655115c53c1afde0da0d0556ad77984da7d0ffa8ab725ea260aa3
    kind: gap
    severity: major
    what: "gxformat2-schema documents that a step's `state` nests conditionals/sections as plain nested YAML (the `state` vs `tool_state` section), but no packaged reference in advance-galaxy-draft-step's, implement-galaxy-tool-step's, or repair-galaxy-draft-topology's bundles documents the companion convention for a step's `in:` connections dict: how to address a nested conditional test-parameter branch or section child as a connection target."
    expected: "Extend gxformat2-schema with an explicit subsection on `in:` key addressing for nested tool state -- the pipe-delimited path convention (e.g. `db_opts|lexicmap_index`, `advanced_settings|align_min_match_pident`) for conditional-branch and section-child parameters -- mirroring the existing `state`-nesting subsection, and include it in implement-galaxy-tool-step's packaged references (the leaf Mold that performs this binding), not just as an assumed convention."
    evidence: "Implementing lexicmap_search (toolshed.g2.bx.psu.edu/repos/iuc/lexicmap/lexicmap_search 0.9.0+galaxy1) required wiring workflow inputs onto a `gx_conditional`'s case-owned parameter (`lexicmap_index`, under `db_opts_selector: db`) and five `gx_section` children of `advanced_settings`. None of this run's loaded, packaged references named or demonstrated the `|`-joined `in:` key form needed to address them; it was inferred from general Galaxy/gxformat2 familiarity outside any packaged bundle content, which the skill's own Runtime Notes direct against ('use only files packaged in this skill bundle and user-supplied artifacts')."
    status: filed
    issue: https://github.com/galaxyproject/foundry/issues/571

  - id: tool-util-cli-toolshed-fetch-rejects-real-filtered-list-collection-output
    raised_by: advance-galaxy-draft-step
    observed_in:
      mold:
        name: advance-galaxy-draft-step
        path: content/molds/advance-galaxy-draft-step/index.md
        revision: 4
        content_hash: c92d452f69a7b40bcab5fc8e63b03e1e2abf98f2a636abc4c93ad7f3c1d03082
        foundry_head: 63a3f9cf9c97a637fe1628d298e524f35709e289
    subject:
      kind: related-project
      label: "galaxy-tool-util-ts / @galaxy-tool-util/cli (gxwf, galaxy-tool-cache) -- ToolShed-fetch-to-ParsedTool decoder"
      locator: "https://github.com/jmchilton/galaxy-tool-util-ts (packages/cli, @galaxy-tool-util/cli@1.8.1)"
    kind: defect
    severity: major
    what: "`galaxy-tool-cache add`/`summarize` (and `gxwf draft-validate`'s own internal tool-state fetch) fail to fetch or parse a real, currently-published IUC Tool Shed wrapper -- toolshed.g2.bx.psu.edu/repos/iuc/kmindex/kmindex_query -- for every galaxy-suffixed version (tried and reproduced for 0.6.0+galaxy1, 0.6.0+galaxy2, 0.6.1+galaxy2, 0.6.1+galaxy3, 0.6.1+galaxy4, 0.6.1+galaxy5), raising a decode error at `outputs[1].structure: is missing` against the upstream parsedToolSchema's collection-output branch. Only the repo's original unsuffixed '0.6.0' version (which predates the wrapper's `<collection type=\"list\"><filter>...</filter><discover_datasets .../></collection>`-shaped output, added by tools-iuc PR #8208 'support multiple indices simultaneously') parses successfully. `add --galaxy-url https://usegalaxy.org` was also tried as a fallback source and fails the same way (plus a second, `inputs is missing` error on that path)."
    expected: "The ToolShed-fetch-to-ParsedTool decoder should populate `structure` (collection_type/discover_datasets/etc.) for a `<collection type=\"list\">` output that declares `discover_datasets` directly and has no `structured_like`/rules -- this is an ordinary, common IUC wrapper shape (also present, filtered by a sibling `<filter>` block, which the schema has no field for at all -- a second, related lossiness worth tracking) -- rather than failing the whole fetch. `gxwf draft-validate` already degrades this failure gracefully to a `skip_tool_not_found` (not a hard fail) for tool-state checking, which is the right fallback behavior for validate; but `galaxy-tool-cache add`/`summarize` themselves give no such degrade and simply cannot produce a manifest for this real, in-production wrapper at all, blocking the normal automated discover -> summarize -> implement path for any workflow step using it."
    evidence: "Reproduced directly this iteration for kmindex_containment_screen (pinned tool_id toolshed.g2.bx.psu.edu/repos/iuc/kmindex/kmindex_query, tool_version 0.6.1+galaxy4, changeset b6fa25b6b436). Root-caused by fetching the wrapper's real upstream XML at the exact matching changeset (Tool Shed API's own `remote_repository_url` for this changeset pointed to github.com/galaxyproject/tools-iuc/tree/main/tools/kmindex; commit 7681be7f40 on that path declares `<tool ... version=\"@TOOL_VERSION@+galaxy4\">`, an exact match) and comparing its `<outputs>` against the successfully-cached bare-'0.6.0' ParsedTool JSON, which lacks the collection output entirely. Worked around this run by hand-reconstructing galaxy-tool-summary.json's `parsed_tool`/`input_schemas` directly from that verified upstream XML rather than via the normal automated `galaxy-tool-cache` path (documented in that file's own `warnings[]`)."
    status: filed
    issue: https://github.com/galaxyproject/foundry/issues/572

  - id: implement-galaxy-tool-step-udt-binding-undocumented
    raised_by: advance-galaxy-draft-step
    observed_in:
      mold:
        name: implement-galaxy-tool-step
        path: content/molds/implement-galaxy-tool-step/index.md
        revision: 9
        content_hash: 197be2d9b27741c3f258092f8392654369d1ca234d7fad492387ff69d656292b
        foundry_head: 63a3f9cf9c97a637fe1628d298e524f35709e289
    subject:
      kind: mold
      label: implement-galaxy-tool-step
      locator: content/molds/implement-galaxy-tool-step/index.md
      content_hash: 197be2d9b27741c3f258092f8392654369d1ca234d7fad492387ff69d656292b
    kind: gap
    severity: minor
    what: "summarize-galaxy-tool's own Inputs section states 'Authored UDTs from author-galaxy-tool-wrapper bypass this Mold,' correctly routing a locally-authored `GalaxyUserTool` around tool-summary generation. But nothing downstream documents what happens next: implement-galaxy-tool-step's declared Inputs list only `galaxy-tool-summary` (never a `galaxy-user-tool-definition`), and its procedure text (step 2, 'Bind to the tool summary') only covers a Tool-Shed-pinned wrapper or a bare/stock built-in id -- there is no documented convention for how a step's `tool_id`/`tool_version`/`state` should be populated when the resolved wrapper is an authored UDT instead of either of those two cases (e.g. whether `tool_id` should mirror the UDT's own `id` field with no `tool_shed_repository` block, by analogy to a bare/stock id, or something else)."
    expected: "Extend implement-galaxy-tool-step's declared Inputs to include the `galaxy-user-tool-definition` artifact as an alternate input when the discover-or-author branch fell through to authoring, and add an explicit procedure step (or sub-case under step 2) for binding a step to a UDT: what `tool_id`/`tool_version` should hold, that no `tool_shed_repository` block applies, and how the UDT's own declared `inputs`/`outputs` names become the step's `in:`/`out:` port names -- mirroring the level of detail already given for the Tool-Shed-pin and bare/stock-id cases."
    evidence: "Authored a GalaxyUserTool wrapper this iteration for a step with a confirmed Tool Shed discovery miss. Absent any documented binding convention, proceeded by analogy: set the step's `tool_id` to the UDT's own `id`, `tool_version` to its `version`, and no `tool_shed_repository` block (treating it like a bare/stock tool id). `gxwf draft-validate --concrete` accepted this shape (`draft valid`, `Concrete: OK`) -- the tool_id triggered a live Tool Shed lookup that 404'd and was gracefully downgraded to a `skip_tool_not_found` (the same non-fatal bucket used for an unrelated tool whose fetch failed for network reasons), not validated against the UDT's own declared contract at all. So the binding convention used happened to pass, but this was inferred from a different case's documented pattern, and draft-validate's accept path gives no positive confirmation that a UDT-backed step's ports/state were checked against anything real."
    status: filed
    issue: https://github.com/galaxyproject/foundry/issues/573

  - id: galaxy-user-tool-authoring-missing-element-identifier-expression
    raised_by: advance-galaxy-draft-step
    observed_in:
      mold:
        name: advance-galaxy-draft-step
        path: content/molds/advance-galaxy-draft-step/index.md
        revision: 4
        content_hash: c92d452f69a7b40bcab5fc8e63b03e1e2abf98f2a636abc4c93ad7f3c1d03082
        foundry_head: 63a3f9cf9c97a637fe1628d298e524f35709e289
    subject:
      kind: research
      label: galaxy-user-tool-authoring
      locator: content/research/galaxy-user-tool-authoring/index.md
      content_hash: 365fad455369945a34bda537c8dbf409a0966f0f88da9fe78b61cff32d5b0d0d
    kind: gap
    severity: minor
    what: "galaxy-user-tool-authoring.md's section 3 ('Expression syntax in shell_command') documents exactly two ways to read a `data` input in a `shell_command`/`configfiles` expression -- a scalar/file path via `$(inputs.NAME.path)` -- and says nothing about a `data` input's other available File-object fields. In particular it never mentions `element_identifier`, even though the installed `@galaxy-tool-util/schema` package's own `gx-data.js` parameter-generation code declares, for the `job_runtime` state representation (the one governing values available inside a running job's command/configfile expressions), a File object shape of `{ class: 'File', basename, location, path, nameroot, nameext, format, size, element_identifier: S.optional(S.String) }` -- i.e. `$(inputs.NAME.element_identifier)` is a real, schema-backed expression, not merely `.path`."
    expected: "Extend galaxy-user-tool-authoring.md section 3 with an explicit line documenting `$(inputs.NAME.element_identifier)` (available when the input is a mapped collection element, matching the classic Galaxy tool-XML idiom already documented elsewhere in this same skill family's convert-nfcore-module-to-galaxy-tool notes: '`$input.element_identifier` in `<command>`'), alongside the existing `.path` documentation -- this is exactly the mechanism an authored UDT needs to preserve a Galaxy collection's per-element identifier into its output content without adding an extra collection-element-identifier-extraction producer step/port to the workflow topology."
    evidence: "Authoring flatten_gene_summary_json_to_row (a GalaxyUserTool mapped one call per gene over a summary_json collection) needed the mapped gene's element_identifier embedded as a literal output column, per this step's own _plan_state ('preserving element_identifier for the join'). galaxy-user-tool-authoring.md's packaged expression-syntax section, read upfront per this skill's own procedure, documents only `.path` and gives no indication `.element_identifier` exists or is valid. Confirmed the field's existence and validity only by grepping the installed `@galaxy-tool-util/cli` npm package's own schema source (`gx-data.js`) directly, outside any packaged skill reference -- exactly the kind of external verification this skill's Runtime Notes direct against ('use only files packaged in this skill bundle and user-supplied artifacts'). Had a different, plausible-but-wrong design been chosen instead (e.g. inserting a `collection_element_identifiers` producer step purely to recover the gene symbol as a wireable parameter), it would have been an avoidable topology change driven by a documentation gap, not a real tool-shape constraint."
    status: filed
    issue: https://github.com/galaxyproject/foundry/issues/468#issuecomment-5733273474

  - id: gxwf-tool-search-underscore-repo-slug-query-zero-hits
    raised_by: advance-galaxy-draft-step
    observed_in:
      mold:
        name: advance-galaxy-draft-step
        path: content/molds/advance-galaxy-draft-step/index.md
        revision: 4
        content_hash: c92d452f69a7b40bcab5fc8e63b03e1e2abf98f2a636abc4c93ad7f3c1d03082
        foundry_head: 63a3f9cf9c97a637fe1628d298e524f35709e289
    subject:
      kind: related-project
      label: "galaxy-tool-util-ts / @galaxy-tool-util/cli (gxwf tool-search) -- Tool Shed query tokenizer/ranker"
      locator: "https://github.com/jmchilton/galaxy-tool-util-ts (packages/cli, gxwf tool-search, @galaxy-tool-util/cli@1.10.1)"
    kind: defect
    severity: minor
    what: "`gxwf tool-search \"collapse_collections\"` (the literal Tool Shed repo-slug for the exact wrapper this iteration needed, toolshed.g2.bx.psu.edu/repos/nml/collapse_collections) returns `No hits for query: collapse_collections` and exits 2. The space-joined equivalent, `gxwf tool-search \"collapse collections\"`, also returns zero hits. Only a differently-worded query -- `gxwf tool-search \"nml collapse\"` (owner name plus one word) or `gxwf tool-search \"Collapse Collection\"` (the tool's display name, not its repo/id) -- surfaces the tool, and even then only as a lower-ranked hit (#3 of 5) in the owner-name case."
    expected: "The query tokenizer/ranker should score a repo-slug-style query (an underscore-joined or space-joined form of the actual `owner/repo` or `tool_id` string) as at least a partial match against that same repo's own `owner/repo` and `tool_id` fields, rather than zero hits -- a query built directly from a known real repo slug is a common, reasonable way to search and should not require the caller to already know the tool's display name or owner to find it."
    evidence: "Reproduced directly this iteration while resolving kmindex_hit_concat's wrapper (same wrapper already pinned earlier in this run as combine_gene_panel_to_bulk_fasta, nml/collapse_collections/collapse_dataset v5.1.0). `gxwf tool-search \"collapse_collections\"` and `gxwf tool-search \"collapse collections\"` both returned no hits; `gxwf tool-search \"nml collapse\"` returned the correct tool as row 3 of 5; `gxwf tool-search \"Collapse Collection\"` returned it as row 1 of 2. Did not block this iteration (the wrapper identity was already known from the draft's own Identity-pinned `tool_id` and a prior same-run resolution), but would block or mislead a cold discover-shed-tool search that started from only a Tool Shed repo slug."
    status: filed
    issue: https://github.com/galaxyproject/foundry/issues/574

  - id: gxwf-draft-validate-json-flag-emits-non-json-diagnostics-on-stdout
    raised_by: advance-galaxy-draft-step
    observed_in:
      mold:
        name: advance-galaxy-draft-step
        path: content/molds/advance-galaxy-draft-step/index.md
        revision: 4
        content_hash: c92d452f69a7b40bcab5fc8e63b03e1e2abf98f2a636abc4c93ad7f3c1d03082
        foundry_head: 63a3f9cf9c97a637fe1628d298e524f35709e289
    subject:
      kind: related-project
      label: "galaxy-tool-util-ts / @galaxy-tool-util/cli (gxwf draft-validate) -- --json report emission"
      locator: "https://github.com/jmchilton/galaxy-tool-util-ts (packages/cli, gxwf draft-validate, @galaxy-tool-util/cli@1.8.1)"
    kind: defect
    severity: minor
    what: "`gxwf draft-validate <file> --concrete --json` writes one or more human-readable diagnostic lines (e.g. `toolshed fetch failed (...) for iuc~kmindex~kmindex_query: <huge inline TS type dump>`, followed by an ASCII tree of the failing schema path) directly to stdout, ahead of the actual JSON report object, whenever a tool-state fetch during the --concrete pass fails to decode (same underlying decode failure as ledger entry `tool-util-cli-toolshed-fetch-rejects-real-filtered-list-collection-output`). This happens even though stderr is a separate stream the process already uses for nothing observed in this run's redirection test, and even though `--help` explicitly describes `--json` as '(Output structured JSON report)'."
    expected: "When `--json` is passed, stdout should contain only the single parseable JSON report object (matching the flag's own documented contract); any fetch-failure/decode-diagnostic text (which draft-validate itself already degrades gracefully into the report's own `skip_tool_not_found` status/errors array) should go to stderr, or be folded into the JSON report's own warnings/errors, not interleaved onto stdout ahead of the JSON payload."
    evidence: "Ran `gxwf draft-validate galaxy-workflow-draft.gxwf.yml --concrete --json 1>/tmp/dv.out 2>/tmp/dv.err` this iteration (implementing join_gene_summary_rows_into_manifest, an unrelated step -- the failing tool-state fetch was for a different, already-implemented step, kmindex_containment_screen/kmindex_query, whose Tool Shed fetch fails per the already-tracked cli defect above). `/tmp/dv.err` was empty; `/tmp/dv.out`'s first bytes were the literal diagnostic text `toolshed fetch failed (...` (confirmed via `head -c 1 | xxd` = `t`, not `{`), not JSON. Had to manually locate the first line starting with `{` and slice from there before `json.loads` would parse the report. The same command without `--json` (plain human-readable mode) is unaffected since it is not claiming to emit machine-parseable output. Non-blocking this iteration only because the diagnostic text happened to precede rather than interleave with the JSON block, and because I fell back to line-scanning; a caller doing a naive `stdout | jq` would hard-fail."
    status: filed
    issue: https://github.com/galaxyproject/foundry/issues/548#issuecomment-5733272931

  - id: paper-to-test-data-no-scoping-or-toolshed-fixture-guidance
    raised_by: paper-to-test-data
    observed_in:
      mold:
        name: paper-to-test-data
        path: content/molds/paper-to-test-data/index.md
        revision: 2
        content_hash: 863d7b503a77fa63449899f124d1669e181fdfd9a72f81f516a219cdc89cb6bb
        foundry_head: 63a3f9cf9c97a637fe1628d298e524f35709e289
    subject:
      kind: mold
      label: paper-to-test-data
      locator: content/molds/paper-to-test-data/index.md
    kind: gap
    severity: minor
    what: "paper-to-test-data's SKILL.md is a single-paragraph procedure ('read freeform-summary, derive test inputs/outputs') with no packaged Load-On-Demand references and no guidance at all for the ordinary case this run hit: the paper's own named datasets (kmindex Logan shards, LexicMap Logan indices) are production/external-service scale and cannot become a fast test fixture as named. Its next-in-chain sibling, find-test-data, packages exactly this guidance -- references/notes/iwc-test-data-conventions.md's 'small is a documented subset of a real source, not a fabricated stand-in' rule, plus an explicit step to search IWC/Tool-Shed fixtures when the source's own named data is the wrong shape/scale. Run head-to-head on the same freeform-summary (this run's phix174 kmindex/LexicMap workflow), paper-to-test-data alone could only restate the summary's already-flagged gaps, while find-test-data's packaged convention led directly to real, reusable, already-committed Tool-Shed test-data (galaxyproject/tools-iuc tools/kmindex/test-data and tools/lexicmap/test-data) that resolved the equivalent structural fixture."
    expected: "paper-to-test-data should either package the same scoping-down / prefer-upstream-test-fixture guidance find-test-data has (so it is not systematically the weaker half of its own declared fallback chain for any workflow with externally-hosted/production-scale reference data), or its own doc should say explicitly when to fall through early (e.g. 'when the paper's cited data is production/external-service scale and not itself a downloadable fixture, this skill's output should mark those inputs resolved: false and the caller should proceed to find-test-data without re-deriving from the same summary') rather than leaving that judgment entirely to the calling harness."
    evidence: "This run (draft-manuscript-galaxy, phase 7): freeform-summary.md names 109 kmindex Logan shards and up to 25 LexicMap Logan indices (Stage B/C) as the workflow's reference-data universe, all hosted only at usegalaxy.org production scale; paper-to-test-data's procedure and references gave no path to anything smaller. find-test-data's iwc-test-data-conventions.md note, read for the same task, pointed directly at checking the pinned tools' own Tool Shed test-data directories, which surfaced real, already-CI-tested small fixtures (tools-iuc kmindex_query.xml test #6 'using register index', lexicmap.xml tests #3/#4/#6) usable as structural smoke-test data for this workflow's kmindex_containment_screen and lexicmap_search steps. Not blocking this run (the harness's own branch protocol tries both skills in order), but paper-to-test-data contributed no incremental value beyond a plain re-read of freeform-summary.md for this workflow's two hardest inputs."
    status: filed
    issue: https://github.com/galaxyproject/foundry/issues/575

  - id: galaxy-workflow-test-plan-schema-fixture-required-for-scalar-typed-params
    raised_by: freeform-summary-to-galaxy-test-plan
    observed_in:
      mold:
        name: freeform-summary-to-galaxy-test-plan
        path: content/molds/freeform-summary-to-galaxy-test-plan/index.md
        revision: 2
        content_hash: 9c1d56e625a5dd26cb7a82082ac8f4eeb7db287a626bf00e73457c8dd491ad6d
        foundry_head: 63a3f9cf9c97a637fe1628d298e524f35709e289
    subject:
      kind: schema
      label: galaxy-workflow-test-plan (JobInput/Fixture $defs)
      locator: "package://@galaxy-foundry/gxwf-foundry#galaxyWorkflowTestPlanSchema"
      content_hash: 325e9cf1bb5e075fa818dc68868647de95389ca8df8581ee8cf1e2be61196e37
    kind: gap
    severity: minor
    what: "JobInput.fixture is required (non-optional) and always resolves to the Fixture $def, whose four fields (storage, location, checksum, provenance) and whose enum for `storage` (remote-url, in-repo, cvmfs-string, generated-toy, unresolved, null) are all phrased in file/collection-fixture terms. Neither the schema's own field descriptions nor any packaged on-demand note (iwc-test-data-conventions.md, planemo-asserts-idioms.md, galaxy-workflow-testability-design.md) says what to do for a JobInput that is a plain typed scalar workflow parameter (int, float, boolean, or a text param carrying a literal default, e.g. this workflow's kmindex_zvalue=6 or tiling_qc_allow_frameshifts=false) rather than a file or collection. iwc-test-data-conventions.md section 5 covers only the CVMFS/.loc 'bare string matching a data-table value' case, which is a different shape again (a reference-data selector, not an arbitrary typed parameter default)."
    expected: "Either make `fixture` on JobInput conditionally optional/nullable-as-a-whole when the input is a plain scalar parameter (not a file/collection/data-table selector), or add an explicit Fixture convention for this case (e.g. storage: null with the literal default value carried in `location`, as this plan did) so different Molds/runs do not each invent their own encoding for the same common situation."
    evidence: "This run's workflow (galaxy-workflow.gxwf.yml) declares 15 workflow inputs, of which 12 are plain typed scalars with literal pinned defaults (kmindex_zvalue, kmindex_threshold, kmindex_output_format, kmindex_fast, lexicmap_top_n_genomes, lexicmap_advanced_all, and the 6 tiling_qc_* params). Each needed a JobInput entry (required by the TestCase schema) whose required `fixture` object has no natural file/location/checksum reading. Absent any documented convention, this plan used `storage: null` with the default value's string form placed in `location`, and `provenance` noting 'workflow default, per galaxy-workflow.gxwf.yml default' -- a reasoned choice, not a documented one."
    status: filed
    issue: https://github.com/galaxyproject/foundry/issues/576

  - id: freeform-summary-to-galaxy-test-plan-silent-on-concrete-draft-and-test-data-refs-inputs
    raised_by: freeform-summary-to-galaxy-test-plan
    observed_in:
      mold:
        name: freeform-summary-to-galaxy-test-plan
        path: content/molds/freeform-summary-to-galaxy-test-plan/index.md
        revision: 2
        content_hash: 9c1d56e625a5dd26cb7a82082ac8f4eeb7db287a626bf00e73457c8dd491ad6d
        foundry_head: 63a3f9cf9c97a637fe1628d298e524f35709e289
    subject:
      kind: mold
      label: freeform-summary-to-galaxy-test-plan
      locator: content/molds/freeform-summary-to-galaxy-test-plan/index.md
      content_hash: 9c1d56e625a5dd26cb7a82082ac8f4eeb7db287a626bf00e73457c8dd491ad6d
    kind: gap
    severity: minor
    what: "The Mold's declared Inputs (freeform-summary, freeform-galaxy-interface, freeform-galaxy-data-flow, iwc-comparison-notes, iwc-exemplar-gxformat2) and its 'Labels and fixtures are assumed, not bound' procedure section both describe this skill as operating only against template-era briefs, directing label_status: assumed and workflow.label_source: interface-brief in every case. Nothing in the Mold's own SKILL.md anticipates or documents the case actually encountered in this run: a concrete gxformat2 workflow draft and resolved test-data-refs artifact already existed in the harness run-state by phase 8 and were handed to this invocation as extra grounding context, with an explicit instruction to keep the plan consistent with them. The Mold gives no guidance on whether `label_source` should then be 'draft' (since a concrete draft actually was read) or 'interface-brief' (since that is this Mold's only documented input class), nor on how `label_status` should reflect a label that was cross-checked against a real draft rather than merely assumed from a brief."
    expected: "Document an optional grounding-input case in this Mold's Inputs/Procedure: when a caller also supplies the concrete workflow draft and/or resolved test-data refs (as changeset-to-galaxy-test-plan and the *-test-to-galaxy-test-plan siblings already do for their own analogous 'carry forward' cases), state explicitly which `label_source` and `label_status` values apply, and how far the plan may rely on those artifacts before that reliance should instead be deferred to implement-galaxy-workflow-test's own workflow-label cross-check."
    evidence: "This run's harness passed galaxy-workflow.gxwf.yml (the concrete 9-step draft) and test-data-refs.json (phase 7's resolved gene E/J test-data scoping) as additional context alongside this Mold's normal declared inputs, with an instruction to avoid contradicting them. Absent any packaged guidance for this hybrid situation, this plan set workflow.label_source: draft and label_status: resolved throughout (since every label was in fact read from and matches the concrete draft's own ids), which is a reasoned but undocumented departure from the Mold's own stated 'assumed'/'interface-brief' default behavior."
    status: filed
    issue: https://github.com/galaxyproject/foundry/issues/577

  - id: tests-format-job-schema-has-no-path-to-an-intermediate-step-input-port
    raised_by: implement-galaxy-workflow-test
    observed_in:
      mold:
        name: implement-galaxy-workflow-test
        path: content/molds/implement-galaxy-workflow-test/index.md
        revision: 8
        content_hash: 966c486ccda2eb1e0f06afa674c5d6b85a7e7bfcc9b1545309e3ef8b1016f931
        foundry_head: 63a3f9cf9c97a637fe1628d298e524f35709e289
    subject:
      kind: schema
      label: "tests-format (Job / TestJob $defs)"
      locator: "package://@galaxy-foundry/gxwf-foundry#testsFormatSchema"
      content_hash: ff0de5041f4c3de0ab7abe9e7555c5a7120665b077f42661461301b0fb26f372
    kind: gap
    severity: major
    what: "The harness's phase-9 task instruction directed constructing a synthetic LexicMap hits TSV and 'wiring it as the lexicmap_results input the test case feeds toward lexicmap_streamer_tiling_qc' so the am3 assertions would be checkable end-to-end. But lexicmap_results is not a top-level workflow input on galaxy-workflow.gxwf.yml -- it is wired to an internal step output (lexicmap_search/out_file). The tests-format schema's `Job` $def (`TestJob.job`) is `additionalProperties: <value-types>` with no structural provision for keying a value to anything but a top-level workflow input label, and the packaged planemo-workflow-test-architecture.md / planemo-asserts-idioms.md notes describe Planemo's `job:` block exclusively in terms of workflow-level inputs. There is no documented (or apparently possible) tests-format mechanism to inject a value onto an intermediate step's input port for a whole-workflow test."
    expected: "Either document explicitly (in tests-format's schema description or in planemo-workflow-test-architecture.md) that a synthetic fixture for an internal step is NOT expressible via `job:` and must instead be pursued via a real registered index/data-table entry or a separate component/tool-level test -- so a Mold is not directed to do something the format cannot express -- or, if Planemo/Galaxy actually has an undocumented mechanism for this (e.g. a `--test_index`-scoped step-parameter override), document and cite it."
    evidence: "This run: after confirming (by reading tests-format.schema.json's Job/TestJob $defs directly) that job keys can only bind workflow-level input labels, test case gene_e_j_am3_diagnostic_synthetic_lexicmap_index in galaxy-workflow.gxwf-tests.yml could not literally wire the synthetic hits TSV onto lexicmap_streamer_tiling_qc's lexicmap_results port. The synthetic fixture was instead staged under test-data/synthetic_am3/ and its expected effects validated by directly executing the real vendored lexicmap_streamer.py script outside of Planemo, with lexicmap_index_selection left as a documented placeholder pending real index construction (galaxy-test-plan.yml unresolved[0]/[1])."
    status: filed
    issue: https://github.com/galaxyproject/foundry/issues/578

  - id: tests-format-schema-silent-on-sample-sheet-column-definitions-shape
    raised_by: implement-galaxy-workflow-test
    observed_in:
      mold:
        name: implement-galaxy-workflow-test
        path: content/molds/implement-galaxy-workflow-test/index.md
        revision: 8
        content_hash: 966c486ccda2eb1e0f06afa674c5d6b85a7e7bfcc9b1545309e3ef8b1016f931
        foundry_head: 63a3f9cf9c97a637fe1628d298e524f35709e289
    subject:
      kind: research
      label: iwc-test-data-conventions (Collection.rows / sample_sheet shape)
      locator: content/research/iwc-test-data-conventions/index.md
      content_hash: 1921e939703444604440e0768caca6e6b834a73dcfa964f450fa081143e008d3
    kind: gap
    severity: minor
    what: "galaxy-workflow.gxwf.yml's primary input, gene_query_panel, is `collection_type: sample_sheet` with 4 optional per-element `column_definitions` (the per-gene LexicMap sensitivity overrides). tests-format.schema.json's `Collection` $def does carry a `rows` field (additionalProperties: column-name -> array) seemingly meant for exactly this, but neither iwc-test-data-conventions.md (this skill's authoritative note on job/input YAML shapes, which documents List/Paired/list:paired/list:list:paired/composite_data in detail) nor any other packaged note mentions `sample_sheet`, `column_definitions`, or `rows` at all, and a corpus grep for these terms across the whole skill bundle returns nothing beyond the bare schema field."
    expected: "Add a worked `sample_sheet` + `rows:` example to iwc-test-data-conventions.md (or a note it references) alongside the existing collection-shape sections, so a Mold does not have to infer the shape from the schema's generic field alone."
    evidence: "galaxy-workflow.gxwf-tests.yml's test case 2 encodes gene_query_panel's per-gene align_min_match_pident/align_min_match_len/seed_min_prefix/min_qcov_per_genome overrides as `rows: {align_min_match_pident: [60.0, null], ...}` alongside `elements:` -- a reasoned best-effort construction against the bare schema field, passing `gxwf validate-tests --workflow` cleanly, but with no corpus precedent to confirm it is actually what Planemo/Galaxy expects at runtime."
    status: filed
    issue: https://github.com/galaxyproject/foundry/issues/579

  - id: galaxy-workflow-frame-comments-missing-position-size-rejected-by-real-galaxy-import
    raised_by: implement-galaxy-workflow-test
    observed_in:
      mold:
        name: implement-galaxy-workflow-test
        path: content/molds/implement-galaxy-workflow-test/index.md
        revision: 8
        content_hash: 966c486ccda2eb1e0f06afa674c5d6b85a7e7bfcc9b1545309e3ef8b1016f931
        foundry_head: 63a3f9cf9c97a637fe1628d298e524f35709e289
    subject:
      kind: related-project
      label: "Galaxy (galaxyproject/galaxy) WorkflowCommentModel vs. this pipeline's gxformat2 `comments: type: frame` authoring"
      locator: "galaxyproject/galaxy: lib/galaxy/model/__init__.py (WorkflowCommentModel), lib/galaxy/managers/workflows.py (_workflow_from_raw_description)"
    kind: gap
    severity: major
    what: "Attempting `planemo test --test_index 1 --install_galaxy galaxy-workflow.gxwf.yml` (a bounded attempt at the second validation step of this phase) failed before any test data was staged or any tool was run: Galaxy's real workflow-import API rejected galaxy-workflow.gxwf.yml with HTTP 400, 'Field required in (\"frame\",\"position\")' and 'Field required in (\"frame\",\"size\")'. The workflow's own `comments:` block (4 `type: frame` entries grouping steps into Stage A/B/C/D2, e.g. `{type: frame, label: 'Stage A — input assembly', title: ..., contains_steps: [...]}`) carries no `position`/`size` fields, which real Galaxy's WorkflowCommentModel requires for any frame-type comment. This means the concrete workflow produced by earlier phases (advance-galaxy-draft-step / freeform-summary-to-galaxy-template, whichever Mold first authored these frame comments) is schema-valid enough to pass this pipeline's own `gxwf draft-validate --concrete` gate but is NOT actually importable into a real Galaxy instance."
    expected: "Whichever Mold/reference documents the gxformat2 `comments: type: frame` shape should require (or default) `position: {x, y}` and `size: {width, height}` fields, and `gxwf draft-validate`/`validate-galaxy-workflow` should be extended to catch this class of real-Galaxy-only requirement before a Planemo run discovers it at import time."
    evidence: "Full traceback captured during this phase's `planemo test` attempt: bioblend.ConnectionError: Unexpected HTTP status code: 400, pydantic_core ValidationError 'WorkflowCommentModel: frame.position Field required, frame.size Field required', raised from galaxy.managers.workflows.build_workflow_from_raw_description -> model.WorkflowComment.from_dict on the first frame comment `{id:0, type:frame, data:{title:'Stage A — input assembly'}, child_steps:[15], label:'Stage A — input assembly'}`. This blocked test case 1 before tool installation/dependency resolution was ever reached, so it was not possible to determine whether the kmindex/lexicmap/UDT tool chain itself would install and run cleanly."
    status: filed
    issue: https://github.com/galaxyproject/foundry/issues/580

  - id: gxwf-validate-connections-flag-crashes-uncaught-on-format2-dict-shaped-step-in
    raised_by: validate-galaxy-workflow
    observed_in:
      mold:
        name: validate-galaxy-workflow
        path: content/molds/validate-galaxy-workflow/index.md
        revision: 5
        content_hash: 74e3743fc376f33c1eb37832ab4869c327d48f87e015c7a38597cf4c4ca939cd
        foundry_head: 79bf5c3ab98eec3a67948d8bdebb9dba7d8a2359
    subject:
      kind: related-project
      label: "galaxy-tool-util-ts / @galaxy-tool-util/cli (gxwf validate --connections) -- normalized-workflow-to-native step builder"
      locator: "https://github.com/jmchilton/galaxy-tool-util-ts (packages/schema, workflow/normalized/toNative.js:429 _extractConnections; packages/connection-validation, graph-builder.js; @galaxy-tool-util/cli@1.10.1)"
    kind: defect
    severity: major
    what: "`gxwf validate galaxy-workflow.gxwf.yml --json --connections` (and the same with `--strict` added) crashes with an uncaught `TypeError: step.in is not iterable` at `toNative.js:429` (`_extractConnections`, called from `_buildToolStep` -> `_buildStep` -> `_buildNativeWorkflow` -> `toNative` -> `_coerceNormalizedNative` -> `buildWorkflowGraph` -> `validateConnectionsReport` -> `buildConnectionReport`), exits 1, and emits no JSON report at all -- not even a `connection_report: null` degrade the way tool-state fetch failures degrade to `skip_tool_not_found`. This workflow's every step's `in:` is an ordinary gxformat2 mapping (`{port_name: source, ...}`, e.g. `in: {input_list: gene_query_panel}`), the standard and only shape gxformat2's own schema documents for step connections (see this same run's `gxformat2-schema-missing-nested-in-key-convention` entry, which independently confirms and cites this `in:` mapping shape). `_extractConnections`'s `for (const stepInput of step.in)` expects `step.in` to already be an array (the native-format shape), so any format2 workflow with a normal dict-shaped `in:` reaching this code path crashes rather than being converted."
    expected: "`_extractConnections` (or its caller) should convert a format2 dict-shaped `step.in` into the array shape it expects before iterating -- the same normalization `toNative` already performs successfully for the plain `gxwf validate` (no `--connections`) path, since that path reads this exact workflow's steps without error. `gxwf validate --connections` should either work on an ordinary format2 workflow or fail gracefully (a caught, reported error in the JSON `connection_report` field, matching the tool-state fetch failure's graceful `skip_tool_not_found` degrade) rather than throwing an uncaught exception that discards the entire report."
    evidence: "Reproduced directly this phase: `gxwf validate galaxy-workflow.gxwf.yml --json --connections` and `gxwf validate galaxy-workflow.gxwf.yml --json --connections --strict` both exit 1 with the identical stack trace, no stdout JSON at all (the process crashes before writing the report object). `gxwf validate galaxy-workflow.gxwf.yml --json --strict` (same workflow, same flags minus `--connections`) exits 0 and produces a full JSON report (5 ok / 0 fail / 4 skip). This skill's own SKILL.md directs using `--connections` 'when tool cache metadata is available and data-shape compatibility matters, especially around collections and map-over' -- squarely this workflow's shape (9 steps, heavy collection map-over across kmindex/lexicmap/UDT steps) -- so the flag's total unusability here is a real, non-hypothetical gap, not an edge case. Worked around by validating without `--connections` and recording collection/map-over shape compatibility as a residual runtime risk instead."
    status: filed
    issue: https://github.com/galaxyproject/foundry/issues/581

  - id: run-workflow-test-no-mechanism-to-install-authored-galaxyusertool-udts
    raised_by: run-workflow-test
    observed_in:
      mold:
        name: run-workflow-test
        path: content/molds/run-workflow-test/index.md
        revision: 6
        content_hash: unavailable-in-cast
        foundry_head: 79bf5c3ab98eec3a67948d8bdebb9dba7d8a2359
    subject:
      kind: mold
      label: run-workflow-test
      locator: content/molds/run-workflow-test/index.md
    kind: gap
    severity: major
    what: "Neither run-workflow-test's own packaged references (planemo.md, planemo-workflow-test-architecture.md, planemo-asserts-idioms.md, galaxy-workflow-invocation-failure-reference.md) nor author-galaxy-tool-wrapper's (which explicitly states its output is 'a single GalaxyUserTool YAML document, not Galaxy XML') document any mechanism by which a locally-authored GalaxyUserTool YAML definition (no Tool Shed presence) is supposed to reach a Planemo-managed Galaxy's toolbox for a workflow test run. `planemo test`'s only documented tool-injection option, `--extra_tools <file|directory>`, was tried against a directory holding this run's 3 UDT YAML files; planemo emits a `<tool_dir dir=\"...\">` entry into the generated tool_conf.xml (confirmed by reading the generated file directly), and Galaxy's toolbox parses that tool_conf.xml, but Galaxy's classic `tool_dir` scanner only auto-discovers XML tool wrappers -- zero log lines anywhere reference any of the 3 UDT ids/files, and the subsequent real workflow-invocation attempt's HTTP 400 explicitly lists `lexicmap_streamer (version 1.0.0)`, `flatten_gene_summary_json_to_row (version 1.0.0)`, and `kmindex_hit_dedup_max_score (version 1.0.0)` among the 'required tools are not installed', proving the toolbox never registered them at all."
    expected: "Either document a real, working path for a GalaxyUserTool YAML to reach a Planemo-managed toolbox for testing (e.g. a `gxwf`/`galaxy-tool-util` command that lowers `class: GalaxyUserTool` to a classic Galaxy tool XML + script directory that `--extra_tools` can actually load, or an alternate Planemo/Galaxy API this skill bundle should cite), or state explicitly in run-workflow-test's own procedure that an authored UDT with no such lowering step is expected to fail tool-shed/toolbox resolution in a real Planemo run and name the concrete blocking condition as `not-run`/`fail`-with-tool-install-modality rather than leaving the caller to discover this empirically."
    evidence: "Phase 11 of this run (draft-manuscript-galaxy). Copied the 3 UDT YAMLs (galaxy-user-tool.yml, galaxy-user-tool-flatten-gene-summary.yml, galaxy-user-tool-kmindex-hit-dedup-max-score.yml) into a scratch directory and passed it via `planemo test --extra_tools <dir> --install_galaxy --test_index 2 ...`. Verified the generated tmp tool_conf.xml contained `<tool_dir dir=\".../udt-tools\" />`; grepped both this run's planemo-test1.log and planemo-test2.log for the 3 UDT ids/filenames/`GalaxyUserTool` and found zero matches anywhere in either log. The subsequent workflow-invocation attempt for test case 2 (`POST /api/workflows/.../invocations` -> HTTP 400) returned `err_msg: \"Workflow was not invoked; the following required tools are not installed: ... lexicmap_streamer (version 1.0.0), flatten_gene_summary_json_to_row (version 1.0.0), kmindex_hit_dedup_max_score (version 1.0.0), ...\"`, confirming the toolbox has no record of these 3 tool ids under any mechanism this run attempted."
    status: filed
    issue: https://github.com/galaxyproject/foundry/issues/582

  - id: galaxy-tool-util-replacement-collection-requires-rows-key-unconditionally
    raised_by: run-workflow-test
    observed_in:
      mold:
        name: run-workflow-test
        path: content/molds/run-workflow-test/index.md
        revision: 6
        content_hash: unavailable-in-cast
        foundry_head: 79bf5c3ab98eec3a67948d8bdebb9dba7d8a2359
    subject:
      kind: related-project
      label: "galaxy-tool-util (galaxy.tool_util.cwl.util.replacement_collection) -- Planemo job-input staging for sample_sheet collections"
      locator: "https://github.com/galaxyproject/galaxy (packages/tool_util, lib/galaxy/tool_util/cwl/util.py, galaxy-tool-util==25.1.2, vendored into planemo==0.75.47)"
    kind: defect
    severity: major
    what: "`galaxy.tool_util.cwl.util.replacement_collection()` (the function Planemo's `stage_in`/`galactic_job_json` path uses to turn a tests-format `job:` Collection value into a Galaxy HDCA-creation payload) contains `if collection_type.startswith(\"sample_sheet\"): kwds[\"rows\"] = value[\"rows\"]` -- an unconditional dict-index (not `.get()`) on the job value's optional `rows` key. tests-format.schema.json's own `Collection` $def does not require `rows` (this workflow's own test case 1, `kmindex_wiring_smoke_generic_fixtures`, intentionally omits it -- 'not meaningful for generic non-phiX174 fixture content' -- and that test file already passed `gxwf validate-tests --workflow ... --json` cleanly with `rows` absent). Any `sample_sheet`-typed Collection job input that legitimately omits `rows` therefore crashes Planemo's staging step with an uncaught `KeyError: 'rows'` before any Galaxy invocation is even created, rather than surfacing as a graceful assertion/staging-problem report."
    expected: "`replacement_collection` should use `kwds[\"rows\"] = value.get(\"rows\", {})` (or omit the key entirely when absent, if Galaxy's collection-create API tolerates a missing `rows`), matching the tests-format schema's own treatment of `rows` as optional for a `sample_sheet` Collection value."
    evidence: "Reproduced directly this phase: `planemo test --install_galaxy --test_index 1 --extra_tools <dir> galaxy-workflow.gxwf.yml` crashed with `execution_problem: \"'rows'\"`, `status: \"error\"`, `invocation_details: null`, `job: null` in the structured `tool_test_output.json` -- i.e. staging failed before any Galaxy invocation existed. Full traceback: `planemo/galaxy/activity.py:441 stage_in -> galaxy/tool_util/client/staging.py:275 stage -> galaxy/tool_util/cwl/util.py:388 galactic_job_json -> :234 replacement_item -> :362 replacement_collection -> KeyError: 'rows'`. Confirmed the isolating variable by contrast: test case 2 (`gene_e_j_am3_diagnostic_synthetic_lexicmap_index`), whose `gene_query_panel` Collection DOES carry a `rows:` block, staged past this exact code path without error in the immediately following run and reached a real (differently-failing) workflow-invocation attempt. Installed package confirmed at `galaxy_tool_util-25.1.2.dist-info` under planemo 0.75.47's own venv."
    status: filed
    issue: https://github.com/galaxyproject/foundry/issues/583

  - id: galaxy-workflow-invocation-check-rejects-previously-installed-tool-as-not-installed
    raised_by: debug-galaxy-workflow-output
    observed_in:
      mold:
        name: debug-galaxy-workflow-output
        path: content/molds/debug-galaxy-workflow-output/index.md
        revision: 5
        content_hash: 100b3f985f8e309092bfd05cf544a8286e0a878ee2df3a155dfebe6fcdb286f0
        foundry_head: 63a3f9cf9c97a637fe1628d298e524f35709e289
    subject:
      kind: research
      label: galaxy-workflow-invocation-failure-reference
      locator: content/research/galaxy-workflow-invocation-failure-reference/index.md
      content_hash: e8115df6ddca447d47d77cb33becda847cdd19b2cbe9924782bfc180b94bb451
    kind: gap
    severity: major
    what: "The note's 'Request-time validation' surface (API error before a useful invocation state exists, e.g. an HTTP 400 'required tools are not installed') gives no guidance on how to tell a genuine per-tool dependency/installation failure apart from a toolbox-state read that is stale relative to the just-completed install/reload cycle. This phase found a concrete, reproducible tell that the note doesn't mention: in this run's test 2, `planemo-test2.log:292` shows `collapse_collections` (providing `collapse_dataset`) was *skipped* for reinstall because its cached status was already 'Installed' -- no fresh clone, no fresh dependency resolution occurred for it this run -- yet the subsequent `POST /api/workflows/.../invocations` 400 (`planemo-test2.log:1775-1796`) still lists `collapse_dataset` among the 'not installed' tools, 54 seconds later, alongside 3 real freshly-cloned Tool Shed tools and 3 never-installable UDTs. A tool with a confirmed pre-existing good install status being rejected is strong evidence the invocation-validation check read stale/incomplete toolbox state rather than a real, current dependency failure for that specific tool -- but nothing in this note documents 'a previously-Installed repository still appearing in the not-installed list' as a diagnostic signal for this class of race, so the classification had to be reconstructed ad hoc from Galaxy install-manager log lines rather than a documented reference path."
    expected: "Add this diagnostic tell to the 'Request-time validation' row (or a new row) in galaxy-workflow-invocation-failure-reference.md: when a tool listed as 'not installed' in a request-time 400 has a repository install-manager log line showing its install was skipped because it was already 'Installed' from a prior/cached run, that is evidence of a toolbox-state read/reload-timing defect rather than a genuine dependency-resolution failure for that tool, and should route debugging toward Galaxy's toolbox-reload/tool-availability-check code path rather than toward the wrapper's conda requirements."
    evidence: "draft-manuscript-galaxy phase 12. `planemo-test2.log:292`: \"Skipping installation of revision 90981f86000f of repository 'collapse_collections' because it was installed with the (possibly updated) revision 90981f86000f and its current installation status is 'Installed'.\" `planemo-test2.log:1775,1796`: `POST /api/workflows/961e7c4742a92de4/invocations HTTP/1.1 400` with `err_msg` listing `collapse_dataset (version 5.1.0)` among 7 'not installed' tools. Last logged `reload_toolbox` control-task cycle completes at `11:50:13,808`; the invocation POST fires at `11:50:58,531`, 45s later -- ruling out an in-flight reload as the immediate cause and pointing instead at how the invocation-validation code path sources its toolbox snapshot."
    status: filed
    issue: https://github.com/galaxyproject/foundry/issues/584

  - id: galaxy-udt-registration-real-endpoint-is-unprivileged-tools-not-dynamic-tools
    raised_by: run-workflow-test
    observed_in:
      mold:
        name: run-workflow-test
        path: content/molds/run-workflow-test/index.md
        revision: 7
        content_hash: 259675e41c65f7622049cc9ef5f64b7d9f7ac5edb005555a09045e747708218c
        foundry_head: 79bf5c3ab98eec3a67948d8bdebb9dba7d8a2359
    subject:
      kind: related-project
      label: "Galaxy core (lib/galaxy/webapps/galaxy/api) -- dynamic-tool registration endpoints"
      locator: "https://github.com/galaxyproject/galaxy (lib/galaxy/webapps/galaxy/api/dynamic_tools.py, lib/galaxy/webapps/galaxy/api/tools.py)"
    kind: gap
    severity: major
    what: "This run's earlier entry (run-workflow-test-no-mechanism-to-install-authored-galaxyusertool-udts) found no way to load a GalaxyUserTool YAML into a Planemo-managed toolbox and, separately, `POST /api/dynamic_tools` against a real production Galaxy instance (usegalaxy.org) returned HTTP 403 ('You must be an administrator to access this feature'). Neither run-workflow-test's nor author-galaxy-tool-wrapper's packaged references mention that Galaxy exposes a SEPARATE, non-admin-gated endpoint for exactly this: `POST /api/unprivileged_tools`, gated only by a `USER_TOOL_EXECUTE` role plus the instance's `enable_beta_tool_formats` config flag (both satisfied by an ordinary usegalaxy.org account). A regular user's own GalaxyUserTool YAML (converted to the JSON body that endpoint expects) registers there successfully and becomes a real, invocable, user-scoped dynamic tool with a `tool_uuid`."
    expected: "author-galaxy-tool-wrapper and/or run-workflow-test should document `POST /api/unprivileged_tools` as the real-Galaxy registration path for an authored GalaxyUserTool (distinct from the admin-only `/api/dynamic_tools`), including its role/config gating (`USER_TOOL_EXECUTE`, `enable_beta_tool_formats`) and that a successful registration returns a `tool_uuid` a workflow step must reference (see the companion entry on gxformat2 not modeling `tool_uuid`) rather than a bare `tool_id` string."
    evidence: "draft-manuscript-galaxy, post-pipeline deploy step. `POST https://usegalaxy.org/api/dynamic_tools` with a GalaxyUserTool-derived body -> HTTP 403 admin-required. `POST https://usegalaxy.org/api/unprivileged_tools` with the same tool content (after fixing a separate lint defect, see the companion `value: 0` entry) -> 200, returning a real `tool_uuid` for each of this run's 3 UDTs (lexicmap_streamer, flatten_gene_summary_json_to_row, kmindex_hit_dedup_max_score), later confirmed resolvable with zero step errors via `GET /api/workflows/{id}/download` once wired into the imported workflow."
    status: filed
    issue: https://github.com/galaxyproject/foundry/issues/585

  - id: galaxy-workflow-import-enforces-undocumented-per-field-length-limit-on-step-doc
    raised_by: freeform-summary-to-galaxy-template
    observed_in:
      mold:
        name: freeform-summary-to-galaxy-template
        path: content/molds/freeform-summary-to-galaxy-template/index.md
        revision: 6
        content_hash: 0a95f8c04465d48131ace523e0b27af53f1ade6e525a9173c5ff1e936ddc3366
        foundry_head: 63a3f9cf9c97a637fe1628d298e524f35709e289
    subject:
      kind: related-project
      label: "Galaxy core -- POST /api/workflows gxformat2 import, step doc/annotation field"
      locator: "https://github.com/galaxyproject/galaxy (workflow import/gxformat2 conversion path)"
    kind: defect
    severity: major
    what: "A real Galaxy instance (usegalaxy.org, 26.1.2.dev0) rejects `POST /api/workflows` with an opaque, tracebackless HTTP 500 when any step's `doc:` (annotation) field exceeds a real, undocumented length limit. Bisected directly against the live endpoint: a step `doc` of 2000 characters imports fine; 2398 characters 500s; the limit sits at exactly 2048 characters. Neither `gxwf draft-validate`/`gxwf validate` (both passed clean, repeatedly, on the same file) nor galaxy-workflow-draft-format's or gxformat2-schema's packaged notes model or enforce any such limit -- this run's `freeform-summary-to-galaxy-template` output (and later advance-galaxy-draft-step iterations folding `_plan_context` rationale into `doc:` per that skill's own 'a resolved step carries no _plan_* fields' rule) produced 5 step `doc` fields well over 2048 characters, all invisible to every static check available in this toolchain, only surfacing as a bare 500 at real-Galaxy import time."
    expected: "gxwf's draft-validate/validate should check step (and workflow-level) `doc`/`annotation` field length against Galaxy's real limit (~2048 chars, worth confirming the exact DB column width upstream) and fail with a clear diagnostic naming the offending step and its length, rather than letting a workflow pass all local validation and then 500 opaquely at real-Galaxy import. Separately, freeform-summary-to-galaxy-template's convention of folding full authoring/discovery provenance into a step's `doc:` field (rather than only in the run's own ledgers) should be revised to keep `doc:` short and put full provenance in open-requirements/feedback ledger entries instead, given this real, silent ceiling."
    evidence: "draft-manuscript-galaxy, post-pipeline deploy step. Direct `POST /api/workflows` bisection against https://usegalaxy.org/api/workflows: doc length 2000 -> 200 OK; 2398 -> 500 'Uncaught exception in exposed API method' (no traceback surfaced to the client); 2048 -> 200 OK. Trimmed 5 oversized step `doc` fields in galaxy-workflow.gxwf.yml (kmindex_hit_dedup_max_score, lexicmap_streamer_tiling_qc, and 3 others) from full authoring-provenance prose down to concise summaries, after which the same workflow imported successfully."
    status: filed
    issue: https://github.com/galaxyproject/foundry/issues/564#issuecomment-5733272414

  - id: galaxy-unprivileged-tools-lint-crashes-on-integer-input-default-value-zero
    raised_by: author-galaxy-tool-wrapper
    observed_in:
      mold:
        name: author-galaxy-tool-wrapper
        path: content/molds/author-galaxy-tool-wrapper/index.md
        revision: 5
        content_hash: cced8f068cd1ce81fefad23a62fdb4d52b3ce805b2231f64a7fe37f258bc1e09
        foundry_head: ac92d2713fb01b07049027e584b8610e07c88d82
    subject:
      kind: related-project
      label: "Galaxy core -- POST /api/unprivileged_tools tool-creation lint (TestsCaseValidation)"
      locator: "https://github.com/galaxyproject/galaxy (unprivileged dynamic-tool creation / tool linting path)"
    kind: defect
    severity: minor
    what: "Galaxy's real tool-creation lint on `POST /api/unprivileged_tools` (invoked via `input_models_for_tool_source` / `TestsCaseValidation`) throws an uncaught, unhelpfully-reported exception ('TestsCaseValidation ... exception is []', no further detail) when a GalaxyUserTool integer input declares `value: 0` as its default -- confirmed by bisecting the tool's own input list live against the real endpoint: identical input with `value: 1` registers successfully, `value: 0` fails every time. galaxy-user-tool-authoring.md (author-galaxy-tool-wrapper's own packaged reference) documents the `value:` field for scalar inputs generally but does not warn that a literal `0` default on an integer input is unsupported."
    expected: "Either fix Galaxy's lint to handle a zero-valued integer default (the underlying bug), or, until fixed, have galaxy-user-tool-authoring.md warn against declaring `value: 0` on an integer/float UDT input and recommend omitting the default (relying on the step's own explicit wired value) when zero is the semantically correct default."
    evidence: "draft-manuscript-galaxy, post-pipeline deploy step. `POST https://usegalaxy.org/api/unprivileged_tools` with the `lexicmap_streamer` UDT (max_internal_stops integer input, `value: 0`) -> HTTP 400, opaque lint exception. Removing the `value: 0` default from that one input (workflow always supplies it explicitly anyway) -> 200 OK, tool registered with a real `tool_uuid`. Re-added the same field with `value: 1` on a throwaway test copy to confirm the value itself (not the field's mere presence) was the trigger."
    status: filed
    issue: https://github.com/galaxyproject/foundry/issues/586

  - id: gxformat2-step-schema-does-not-model-tool-uuid-for-dynamic-tool-resolution
    raised_by: freeform-summary-to-galaxy-template
    observed_in:
      mold:
        name: freeform-summary-to-galaxy-template
        path: content/molds/freeform-summary-to-galaxy-template/index.md
        revision: 6
        content_hash: 0a95f8c04465d48131ace523e0b27af53f1ade6e525a9173c5ff1e936ddc3366
        foundry_head: 63a3f9cf9c97a637fe1628d298e524f35709e289
    subject:
      kind: related-project
      label: "gxformat2 schema / Galaxy workflow import -- dynamic-tool step resolution"
      locator: "https://github.com/galaxyproject/gxformat2 (step schema), https://github.com/galaxyproject/galaxy (native .ga workflow-step tool_uuid resolution)"
    kind: gap
    severity: major
    what: "A gxformat2 workflow step referencing a real, registered user-scoped dynamic tool (via `POST /api/unprivileged_tools`, see the companion endpoint entry) by bare `tool_id` string alone imports and round-trips (`gxwf convert`) without ever populating or requiring a `tool_uuid` -- but real Galaxy's step-to-tool resolution for a dynamic/unprivileged tool requires the native `.ga` JSON step to carry that tool's real `tool_uuid` explicitly; a bare `tool_id` never resolves to a user-scoped dynamic tool even after it is successfully registered and even though the same `tool_id` string is correct. This left all 3 UDT steps showing 'Tool is not installed' after a clean gxformat2 import, with no schema-level signal of what was missing. Compounded by a second, already-ledgered gxwf defect (`step.in is not iterable` on `gxwf convert`), which blocked using gxwf itself to do the format2->native conversion+injection, forcing a workaround via Galaxy's own `/api/workflows/{id}/download` to get native JSON, hand-inject the 3 real `tool_uuid`s, and re-`POST` that native form."
    expected: "Either extend the gxformat2 step schema/galaxy-workflow-draft-format to model an optional `tool_uuid` field (populated once a dynamic/UDT tool is registered) so a template/draft can carry it through the normal Foundry pipeline, or document explicitly in freeform-summary-to-galaxy-template / advance-galaxy-draft-step / run-workflow-test that a workflow step resolving to a GalaxyUserTool-authored dynamic tool must, at real-Galaxy deployment time, be converted to native `.ga` and have its `tool_uuid` injected post-registration -- this is not currently documented anywhere in the pipeline's packaged references."
    evidence: "draft-manuscript-galaxy, post-pipeline deploy step. gxformat2 import of galaxy-workflow.gxwf.yml (all 3 UDT tool_ids correct, tools already registered via /api/unprivileged_tools) still showed all 3 steps as tool-unresolved in the imported workflow. Fetched the imported workflow natively via `GET /api/workflows/{id}/download`, confirmed each UDT step's JSON had no `tool_uuid` key at all; manually set `tool_uuid` to each of the 3 registered tools' real UUIDs in that native JSON and re-imported via `POST /api/workflows` (native form) -- resulting workflow (id 575e9ee747b2031f) shows zero step errors on the same `GET .../download` check."
    status: filed
    issue: https://github.com/galaxyproject/foundry/issues/587

  - id: lexicmap-search-sample-sheet-column-to-scalar-port-defect
    raised_by: debug-galaxy-workflow-output
    observed_in:
      mold:
        name: debug-galaxy-workflow-output
        path: content/molds/debug-galaxy-workflow-output/index.md
        revision: 5
        content_hash: 100b3f985f8e309092bfd05cf544a8286e0a878ee2df3a155dfebe6fcdb286f0
        foundry_head: 63a3f9cf9c97a637fe1628d298e524f35709e289
    subject:
      kind: mold
      label: advance-galaxy-draft-step
      locator: content/molds/advance-galaxy-draft-step/index.md
      content_hash: c92d452f69a7b40bcab5fc8e63b03e1e2abf98f2a636abc4c93ad7f3c1d03082
    kind: defect
    severity: major
    what: "advance-galaxy-draft-step's phase 6 iteration 2 (lexicmap_search implementation) wired the workflow's four per-gene LexicMap sensitivity overrides (align_min_match_pident, align_min_match_len, seed_min_prefix, min_qcov_per_genome) directly from `gene_query_panel` (a `sample_sheet` collection whose per-element columns carry these values) onto lexicmap_search's plain scalar `advanced_settings|*` tool ports (`in: advanced_settings|align_min_match_pident: gene_query_panel`, etc.) -- a binding that can never work. A sample_sheet collection's per-element column values are only readable by a tool that itself declares a `data_collection` input accepting `sample_sheet` and reads columns internally via `DatasetCollectionWrapper.sample_sheet_row()` in its own Cheetah/XML template; `lexicmap_search` (iuc/lexicmap 0.9.0+galaxy1) is an ordinary IUC tool with plain float/int scalar parameters and no sample_sheet awareness whatsoever. There is no generic Galaxy workflow-wiring mechanism to bind an arbitrary other tool's scalar parameter to a sample_sheet column at runtime -- rewiring the *connection* alone can never fix this; the whole design of carrying these 4 values as sample_sheet columns was unworkable from the start for feeding a non-sample_sheet-aware tool. Nothing in this pipeline's static checks caught it: `gxwf draft-validate`/`gxwf validate` both passed clean repeatedly on this file (structural validation OK, 0 fail/0 structure_errors), the deploy step's own `GET /api/workflows/{id}/download` check showed zero step errors, and `gxwf validate --connections` (which would in principle check collection-algebra/map-over connection-type compatibility) cannot run at all against this file due to an already-ledgered, unrelated crash (`gxwf-validate-connections-flag-crashes-uncaught-on-format2-dict-shaped-step-in`). The defect was only exposed by a real invocation on live usegalaxy.org: both per-gene lexicmap_search jobs (invocation `7a25084290281d4f`, jobs `bbd44e69cb8906b5703a6db6ea90ddf0` and `bbd44e69cb8906b55890e15ae6aa2602`) errored pre-execution -- no command line, no stdout/stderr, no exit code -- with the job's actual submitted `advanced_settings` tool_state literally holding `\"<galaxy.model.DatasetCollectionElement(99373144) at 0x7f1f8adf6cb0>\"` (the raw Python object repr) in place of a resolved float/int for all four ports."
    expected: "implement-galaxy-tool-step / advance-galaxy-draft-step's packaged references should document, as a hard rule, that a `sample_sheet` collection's per-element columns cannot be wired directly onto another (non-sample_sheet-aware) tool's plain scalar parameter -- only onto a tool that itself declares a `data_collection` input parameter for that sample_sheet. For the common case of needing a per-element-varying scalar value to feed an ordinary tool's parameter, the correct, real Galaxy pattern (verified working here) is: a parallel `list` collection (not `sample_sheet`) whose element_identifiers match the driving collection's, one dataset element per gene holding that gene's real value as plain text, consumed by a Galaxy core `param_value_from_file` (0.1.0) step mapped over it -- note this tool exposes one output port per `param_type` (`text_param` / `integer_param` / `float_param` / `boolean_param`; the packaged references should say to use the port matching the chosen `param_type`, not a generic `output`), whose result then wires into the target tool's scalar port. Separately, `gxwf validate --connections` should (once its unrelated crash is fixed) flag a sample_sheet-to-scalar-parameter connection as a real type error, since it is never valid regardless of collection contents."
    evidence: "draft-manuscript-galaxy, phase 12 (debug-galaxy-workflow-output) + live remediation. Root-caused via `GET /api/jobs/{id}?full=true` on both failed lexicmap_search jobs under invocation `7a25084290281d4f` (history `bbd44e69cb8906b51b4da6f15800819c`), showing the literal `DatasetCollectionElement` repr in `advanced_settings.align_min_match_pident`/`align_min_match_len`/`seed_min_prefix`/`min_qcov_per_genome`, while `db_opts.lexicmap_index` (a real ordinary text-select port) resolved correctly to `\"Viral\"` in the same job -- isolating the fault to the sample_sheet-column ports specifically. Confirmed the real fix mechanism by inspecting `GET /api/tools/param_value_from_file?io_details=true` (four per-param_type output ports, not a single generic one) and rebuilt the binding for real: 4 new real `list` collections created via `POST /api/dataset_collections` (`gene_align_min_match_pident_panel` id `5f39d50dafb0838b`, `gene_align_min_match_len_panel` id `ae22378905d7f2b7`, `gene_seed_min_prefix_panel` id `1ea756628560cfef`, `gene_min_qcov_per_genome_panel` id `7565b2b9a270b623`; each E/J, values 60.0/70.0, 35/50, 15/17, 30.0/0.0 respectively -- J using the wrapper's own confirmed tool defaults 70.0/50/17, min_qcov_per_genome having no real wrapper default so 0.0 (no-op floor) was used deliberately for genes without a production override), plus 4 new `extract_<name>_per_gene` (`param_value_from_file` 0.1.0) steps, rewiring lexicmap_search's 4 broken `in:` ports onto these steps' typed output ports instead of `gene_query_panel`. Fix applied to galaxy-workflow.gxwf.yml (local source of truth) and re-validated clean (`gxwf validate`: 9 ok / 0 fail / 4 skip_tool_not_found, all 4 skips pre-existing UDT/toolshed-fetch gaps unrelated to this change)."
    status: filed
    issue: https://github.com/galaxyproject/foundry/issues/588

  - id: galaxy-workflow-update-exact-tools-defaults-true-and-discards-per-step-error-detail
    raised_by: debug-galaxy-workflow-output
    observed_in:
      mold:
        name: debug-galaxy-workflow-output
        path: content/molds/debug-galaxy-workflow-output/index.md
        revision: 5
        content_hash: 100b3f985f8e309092bfd05cf544a8286e0a878ee2df3a155dfebe6fcdb286f0
        foundry_head: 63a3f9cf9c97a637fe1628d298e524f35709e289
    subject:
      kind: related-project
      label: "Galaxy core -- PUT /api/workflows/{id} (update_workflow_from_raw_description, WorkflowUpdateOptions)"
      locator: "https://github.com/galaxyproject/galaxy (lib/galaxy/webapps/galaxy/api/workflows.py, lib/galaxy/managers/workflows.py, lib/galaxy/workflow/modules.py)"
    kind: defect
    severity: major
    what: "`PUT /api/workflows/{id}` resolves every step's tool via `trans.app.toolbox.get_tool(tool_id, tool_version=tool_version, exact=exact_tools, tool_uuid=tool_uuid)`, where `exact_tools` (a field on the real `WorkflowUpdateOptions` Pydantic model) defaults to `True` unless the request body explicitly sets it -- the exact same false-positive tool-resolution behavior already observed at invocation time (workaroundable there via `require_exact_tool_versions: false`) recurs at save time with a DIFFERENT, undocumented field name (`exact_tools`, not `require_exact_tool_versions`) and no obvious hint that a save-time escape hatch even exists. Worse: when `get_tool(..., exact=True)` fails for one or more steps, `update_workflow_from_raw_description` builds a real, specific per-step message (`f\"Step {n+1}: Requires tool '{tool_id}'.\"`) into `missing_tool_tups` and raises `MissingToolsException`, but the API handler in `workflows.py` unconditionally rewrites this to a generic `{\"err_msg\": \"This workflow contains missing tools. It cannot be saved until they have been removed from the workflow or installed.\", \"err_code\": 0}` -- discarding the specific, already-computed per-step list before it ever reaches the client, even though every one of the flagged tools was independently confirmed installed at exactly the pinned tool_id/tool_version via `/api/tools/{id}/build`."
    expected: "(1) `PUT /api/workflows/{id}`'s error response should surface the specific per-step missing-tool list it already computed internally (`missing_tool_tups`) instead of discarding it for a generic message with `err_code: 0` -- this alone would have cut a full diagnostic round-trip. (2) Document `exact_tools`/`allow_missing_tools` (WorkflowUpdateOptions' real fields) as the save-time equivalent of invocation-time's `require_exact_tool_versions`/`allow_tool_state_corrections`, ideally in the same place, since a caller who already learned about one escape hatch has no way to guess the other exists under a different name for a different endpoint."
    evidence: "draft-manuscript-galaxy, post-pipeline deploy step. `PUT /api/workflows/575e9ee747b2031f` with a corrected native workflow JSON (bare `{\"workflow\": {...}}` body) -> `{\"err_msg\": \"This workflow contains missing tools...\", \"err_code\": 0}` for 4 tools (collapse_dataset 5.1.0, lexicmap_search 0.9.0+galaxy1, kmindex_query 0.6.1+galaxy4, collection_column_join 0.0.3) independently re-confirmed installed via `/api/tools/{id}/build` moments before and after. Re-submitting the identical `workflow` content with two added top-level sibling keys, `{\"exact_tools\": false, \"allow_missing_tools\": true}`, succeeded immediately with no other change."
    status: filed
    issue: https://github.com/galaxyproject/foundry/issues/589

  - id: author-galaxy-tool-wrapper-no-check-of-wrapped-scripts-own-declared-input-ordering-assumption
    raised_by: debug-galaxy-workflow-output
    observed_in:
      mold:
        name: debug-galaxy-workflow-output
        path: content/molds/debug-galaxy-workflow-output/index.md
        revision: 5
        content_hash: 100b3f985f8e309092bfd05cf544a8286e0a878ee2df3a155dfebe6fcdb286f0
        foundry_head: 63a3f9cf9c97a637fe1628d298e524f35709e289
    subject:
      kind: mold
      label: author-galaxy-tool-wrapper
      locator: content/molds/author-galaxy-tool-wrapper/index.md
      content_hash: cced8f068cd1ce81fefad23a62fdb4d52b3ce805b2231f64a7fe37f258bc1e09
    kind: gap
    severity: major
    what: "This is NOT a report of a bug in the wrapped script itself (that lived entirely in the user's own private nekrut/disassembler repository and was fixed there directly, with a patch prepared for the user's own separate upstream review -- out of scope for this ledger). The Foundry-relevant gap is in the authoring process: author-galaxy-tool-wrapper wrapped `lexicmap_streamer.py` as the `lexicmap_streamer` GalaxyUserTool without ever checking whether the script's own explicit, self-declared input-ordering assumption actually holds for the real upstream Galaxy tool this step would be wired to consume from in the concrete workflow. The vendored script's own code comment stated the assumption outright ('the stream is grouped by accession one block at a time, so all of an accession's HSPs must be contiguous... detect a reappearance and stop') -- a directly inspectable, static signal that this script's correctness depends on a specific ordering property of its input. Nothing in author-galaxy-tool-wrapper's packaged authoring/review process prompts for cross-checking a wrapped script's own declared input-shape assumptions against the actual, real output ordering of the specific upstream Tool Shed tool (`iuc/lexicmap/lexicmap_search`) it is wired to consume from later in the same workflow (implement-galaxy-tool-step's job, but with no input from authoring about what to check). The gap was only caught because a live, real, production-scale invocation happened to exercise it -- run-workflow-test's own synthetic 3-decoy fixture never could have, since a 3-element fixture is trivially 'grouped' by construction regardless of the bug."
    expected: "author-galaxy-tool-wrapper's review/authoring procedure should add an explicit check step: when a wrapped script's own source/docstring/comments assert an assumption about the shape, ordering, or grouping of its input data (a `sort`/`group`/`contiguous`/'assumes' style comment is a strong, mechanically-greppable signal), the authoring pass should either (a) verify that assumption against the real, documented output-ordering behavior of the specific upstream tool the step will actually be wired to (not just against a hand-built test fixture that can't exercise the failure mode), or (b) explicitly flag the assumption as an open, unverified risk in the step's `doc:`/ledger for a later phase (e.g. run-workflow-test or a dedicated real-data smoke test) to confirm before the workflow is treated as production-ready. A synthetic fixture alone -- however carefully constructed -- cannot substitute for this check when the fixture is, by construction, too small to violate the assumption being tested."
    evidence: "draft-manuscript-galaxy. `lexicmap_streamer.py`'s own committed comment (vendored verbatim into galaxy-user-tool.yml's configfiles entry at authoring time) explicitly named the exact risk that later caused two live job failures on real usegalaxy.org production data (invocation 802eff260023dc62, both lexicmap_streamer_tiling_qc jobs) against the real, correctly-functioning `iuc/lexicmap/lexicmap_search` 0.9.0+galaxy1 output (confirmed via direct byte-range inspection: real LexicMap search output is ranked by match quality across all matched genomes, not grouped by target accession -- the assumption was false for this specific real upstream tool from the start). The 3-decoy synthetic fixture used at test-authoring time (test-data/synthetic_am3/*.tsv) could not have surfaced this: a 3-row/3-accession fixture cannot exhibit a 'reappearing accession' pattern by construction, regardless of whether the underlying assumption holds for real data."
    status: filed
    issue: https://github.com/galaxyproject/foundry/issues/590

  - id: ga-download-normalizes-tool-id-producing-workflows-that-save-but-cannot-invoke
    raised_by: manual-deploy
    observed_in: null
    subject:
      kind: research
      label: galaxy-workflow-invocation-failure-reference
      locator: content/research/galaxy-workflow-invocation-failure-reference/index.md
      content_hash: null
    kind: defect
    severity: major
    what: "`GET /api/workflows/{id}/download?style=ga` emits each Tool Shed step's `tool_id` as the UNVERSIONED repository path (`toolshed.g2.bx.psu.edu/repos/iuc/kmindex/kmindex_query`) with the version carried separately in `tool_version`. Re-importing that exact document -- the download/edit/upload round trip that any external deploy script performs -- produces a workflow that SAVES successfully, shows `errors: null` on every step via `GET /api/workflows/{id}?legacy=false`, and lints clean, but CANNOT be invoked: `POST /api/workflows/{id}/invocations` returns HTTP 400 `Workflow was not invoked; the following required tools are not installed: <tool> (version <v>)` naming tools that are installed at exactly that id and version. Setting `tool_id` to the full VERSIONED path (`.../kmindex_query/0.6.1+galaxy4`) with no other change makes the same workflow invoke immediately. The round trip is therefore lossy in a way that is invisible until invocation: the format the server hands you back is not a format the server will accept for execution. This is distinct from the existing entry `galaxy-workflow-invocation-check-rejects-previously-installed-tool-as-not-installed`, which describes a STALE-TOOLBOX-READ race under planemo with a reload-timing signature; this defect is deterministic, has no timing component, and reproduces on demand against a long-warm production toolbox."
    expected: "(1) `POST /api/workflows/{id}/invocations` should resolve an unversioned `tool_id` together with the step's own `tool_version` field, exactly as the save path and the editor already do -- or, failing that, the error should say the id is unversioned rather than claiming an installed tool is not installed, which routes debugging toward tool installation instead of toward id formatting. (2) `download?style=ga` should round-trip losslessly: either emit the versioned id, or the invocation check should accept what the download emits. As it stands, the documented way to fetch a workflow produces a document that silently loses executability."
    evidence: "draft-manuscript-galaxy, paper-scale rerun setup, 2026-09-18. ISOLATED on a purpose-built 1-step probe workflow (single `tp_cat` step, plain `POST /api/workflows` body `{\"workflow\": ...}` with NO `exact_tools`/`allow_missing_tools` keys, so those flags are excluded as a cause): `tool_id: .../text_processing/tp_cat` + `tool_version: 9.11+galaxy0` -> invocation 400 `required tools are not installed: .../tp_cat (version 9.11+galaxy0)`; changing ONLY `tool_id` to `.../text_processing/tp_cat/9.11+galaxy0` -> invocation scheduled, jobs ran to `ok`. CORROBORATED on the real workflow 575e9ee747b2031f: version 4 (unversioned ids, from a `download?style=ga` round trip) saved fine with `steps with errors: 0` and every tool independently confirmed present via `GET /api/tools/{versioned-id}`, yet invocation returned 400 listing collapse_dataset 5.1.0, kmindex_query 0.6.1+galaxy4, lexicmap_search 0.9.0+galaxy1 and collection_column_join 0.0.3 as not installed; version 5, identical apart from versioned `tool_id`s, invoked successfully (invocation 6372d41d2d9cf1c6)."
    status: filed
    issue: https://github.com/galaxyproject/foundry/issues/591

• Galaxy Data-Flow Design Brief — Workflow A (SRA Landscape & Spike-In Sieve)
• 0. Scope (user-confirmed, binding for this run)
• 1. Workflow-level inputs (Workflow A)
• 2. Nodes and edges
• 3. Collection idiom summary
• 4. Unresolved Galaxy tool needs
• 5. Placeholder (shape-changing) transformations
• 6. Confidence summary
• 7. Open questions carried forward (not resolved by this brief)
• 8. Open-requirements ledger cross-reference
• 9. Foundry feedback

# Galaxy Data-Flow Design Brief — Workflow A (SRA Landscape & Spike-In Sieve)

Source: `freeform-summary.md` (Stages A, B, C, D2) and `freeform-galaxy-interface.md` §1
(Workflow A). This brief is an abstract Galaxy-facing DAG for **Workflow A only** — inputs,
nodes, edges, collection map/reduce points, shape-changing placeholders, and unresolved tool
needs. It is not `gxformat2` and does not resolve exact Tool Shed changesets; that is
`freeform-summary-to-galaxy-template` / `implement-galaxy-tool-step`'s job.

## 0. Scope (user-confirmed, binding for this run)

This run builds **only Workflow A**: kmindex containment screen → LexicMap streaming search →
the custom multi-HSP tiling/QC/haplotype-collapsing step the user calls **LexicMapStreamer**
(built on `stream_lexicmap_msa.py`) → the am3 spike-in diagnostic. Workflows B, C, D from the
interface brief are **not wired here**:

- **Workflow B** (Stage E — Idaho 2024 / Dickins 2009 experimental-evolution trajectory
  validation, bowtie2/samtools/mpileup) is a **confirmed nice-to-have**, explicitly scoped as a
  **separate, later workflow** — out of scope for this run, not abandoned. No data-flow nodes
  are proposed for it here; see `open-requirements.ledger.yml` entry
  `workflow-scope-boundary-unresolved` (now resolved with this scope decision recorded).
- **Workflows C and D** (Stage C′ disassembler/logan-walker cDBG track, Stage F/G/H VEP +
  dual-coding calibration, Stage I ChronAeon) are out of scope for this run's design entirely.
  Their ledger entries are carried forward untouched below.

## 1. Workflow-level inputs (Workflow A)

| Input | Shape | Feeds |
|---|---|---|
| `Reference genome (NC_001422.1)` | `data` (fasta) | **Not wired to any node below** — see §7 open question; likely provenance/documentation only, not a computational dependency of Stages B/C as described in the source |
| `Gene query panel` | `data_collection`, `sample_sheet` (10 elements A,B,C,D,E,F,G,H,J,K), `column_definitions` carrying per-gene LexicMap sensitivity overrides (`align_min_match_pident`, `align_min_match_len`, `seed_min_prefix`, `min_qcov_per_genome`, all `optional: true`) | N1 (combine), N4 (per-gene LexicMap search) |
| `Logan kmindex DB selection` | `string` (multi-select over the 109 hardcoded shard names) | N2 |
| `LexicMap index selection` | `string` (multi-select over 25 domain indices / 5-index `TARGETED_PHAGE_INDICES` subset) | N4 |
| `kmindex parameters` | typed params (`zvalue=6`, `threshold=0.3`, `format=json`, `fast=false`) | N2 |
| `LexicMap parameters` | typed params (`top_n_genomes=0`, `advanced_settings\|all=true`) | N4 (default branch) |
| `Tiling/QC parameters` | typed params (`min_coverage=0.80`, `min_coverage_partial=0.50`, `min_pident=60.0`, `max_internal_stops=0`, `sample_cap=10`, `allow_frameshifts` boolean, default unconfirmed) | N5 |
| `Nominal-taxonomy BioProject classification table` | `data` (tabular, reference) | **Not wired to any node** — modeled as reference/provenance data only, per ledger `nominal-taxonomy-audit-not-modeled-as-step` |

## 2. Nodes and edges

**N1 — Combine per-gene panel into bulk query FASTA** (fan-in / combine)
Input: `Gene query panel` (sample_sheet, 10 elements) → Output: one combined multi-FASTA dataset.
Idiom: fan-in/combine (`fan-in-bundle-consume-and-flatten` family) — collection-of-sequences to
one dataset. Confidence: high (source states the combined panel is the literal kmindex query
input).

**N2 — kmindex containment screen** (map-over)
Input: N1 output (single FASTA) × `Logan kmindex DB selection` → Output: `kmindex containment
hits (per shard)`, `list` collection of JSON, one element per selected shard.
Tool: real Tool Shed wrapper, `toolshed.g2.bx.psu.edu/repos/iuc/kmindex/kmindex_query/0.6.1+galaxy4`
(existence high confidence). Open: exact Galaxy-side mechanism for exposing 109 DB shard names
as a map-over axis (repeat vs. multi-select vs. per-element data input) — ledger
`kmindex-lexicmap-index-selection-mechanism`.

**N3 — Merge/union kmindex hit maps** (fan-in, custom aggregation)
Input: N2 output (list of per-shard JSON) → Output: `kmindex accession union` (`data`, txt).
Idiom: fan-in reduce, but the merge logic (dedup accessions across shards, keep max containment
score) is currently ad hoc Python (`harvest_collection()` walking the Galaxy history-contents API
32-threaded) — no built-in Galaxy collection operation performs this; flagged as an **unresolved
tool need** (§4). Confidence: medium — transform intent is clear, concrete Galaxy-native
implementation is not.

**N4 — LexicMap streaming search** (nested map-over: gene × index, then per-gene reduce)
Input: `Gene query panel` (sample_sheet, per-gene FASTA + optional per-gene sensitivity columns) ×
`LexicMap index selection` → per-(gene, index) tabular hit table → reduced over the index axis
→ Output: `LexicMap results (per gene)`, `sample_sheet` (mirrors the 10-gene panel).
Tool: real Tool Shed wrapper, `toolshed.g2.bx.psu.edu/repos/iuc/lexicmap/lexicmap_search/0.9.0+galaxy1`
(existence high confidence).
Conditional: for genes A/C/E/K, the sample_sheet's per-gene columns
(`align_min_match_pident=60.0`, `align_min_match_len` 35–50, `seed_min_prefix` 15–17,
`min_qcov_per_genome=30.0`) override the workflow-level `LexicMap parameters` default
(`top_n_genomes=0`, `advanced_settings|all=true`) used by the other six genes. Idiom:
`conditional-transform-or-pass-through` (pick_value between column-supplied override and
workflow default, gated on whether the optional column is populated).
Idiom for the index-axis reduce: `collection-flatten-after-fanout` (list:list → list, preserving
gene `element_identifier`). Per the sample_sheet note, this map-over does **not** propagate
`column_definitions` automatically — the per-gene tuning metadata must be read at the point of
the N4 tool call itself (from the *input* sample_sheet), not assumed to survive onto the output
collection for N5 to reuse.
Confidence: high on tool identity and the two named parameter sets; medium on the exact
index-selection surfacing mechanism (same ledger entry as N2) and on the pick_value wiring for
the conditional per-gene override.

**N5 — LexicMapStreamer: multi-HSP tiling, codon QC, haplotype collapsing** (map-over, multi-output per element)
Input: `LexicMap results (per gene)` (N4 output, sample_sheet) + `Tiling/QC parameters` →
mapped one tool call per gene, each producing **8 files**: `.clean.msa.fasta`,
`.clean.haplotypes.tsv`, `.clean.accessions.fasta`, `.clean_expanded.accessions.fasta`,
`.flagged.accessions.fasta`, `.flagged.tsv`, `.cohort_ledger.tsv`, `.summary.json`.
Outputs map onto the interface brief's: `Clean full-length CDS haplotypes (per gene)`, `Clean
haplotype counts (per gene)`, `Flagged accessions + audit reasons (per gene)`, `Per-accession
cohort ledger (per gene)`, `Per-gene ingestion summary` — all `sample_sheet`-shaped, gene-keyed.
Idiom: map-over with a **record-shaped per-element output**. Because each gene element produces
8 named, heterogeneously-typed artifacts (not one), `sample_sheet:record` (named typed slots) is
a better-fitting output shape than eight parallel `sample_sheet` collections that all have to be
kept in identifier lockstep by hand — flagged as a **design recommendation for the template
phase**, not a decided wiring (medium confidence; the source never states a Galaxy collection
type, this is this brief's translation).
Tool need: **LexicMapStreamer** — see §4, this is the central tool-needs item for Workflow A.
Confidence: high on operation shape and per-gene multiplicity (directly and precisely described
in the source, including the exact 8 output artifacts and the paper's headline Table 1 numbers);
medium-low on the exact CLI-flag-to-Galaxy-input mapping pending wrapper authoring.

**N6 — Ingestion manifest aggregation** (fan-in, tabular bridge)
Input: `Per-gene ingestion summary` (`.summary.json`, sample_sheet, from N5) → Output: `Ingestion
manifest (all genes)` (`data`, csv).
Idiom: tabular bridge / fan-in, akin to `tabular-concatenate-collection-to-table` but the source
rows are JSON, not TSV — each element's `.summary.json` must first be flattened into one row
before concatenation. No generic Galaxy tool flattens an arbitrary per-gene JSON summary into a
manifest row; flagged as a **placeholder transformation** (§5). Confidence: high on intent
(source names `lexicmap_9genes_ingestion_manifest.csv` explicitly as this aggregation), medium on
mechanism.

**N7 — Gene E am3 quarantine audit extraction** (collection-unbox / extract-by-identifier)
Input: `Per-gene ingestion summary` or `Flagged accessions + audit reasons (per gene)` (N5 output,
sample_sheet) → Output: `Gene E am3 quarantine audit` (`data`, JSON, single dataset).
Idiom: `collection-unbox-singleton` (`__EXTRACT_DATASET__`, `which: first` / by
`element_identifier == "E"`). The source states the am3 detection logic lives *inside*
LexicMapStreamer's QC step (not a separate computation), so this node is presented as an
**extraction of the Gene E element**, not a new statistic. Confidence: medium — the source
confirms the am3 diagnostic is computed inside N5's QC pass and is summarized in
`gene_e_am3_quarantine_audit.json`, but does not confirm whether that file is literally the Gene
E element of the `.summary.json` sample_sheet or a ninth, separate per-gene artifact N5 also
emits. Carried as an open question (§7) rather than guessed.

## 3. Collection idiom summary

| Node | Idiom | Pattern reference |
|---|---|---|
| N1 | fan-in / combine | `fan-in-bundle-consume-and-flatten` |
| N2 | map-over (list axis over DB shard selection) | `manifest-to-mapped-collection-lifecycle` |
| N3 | fan-in / custom merge (no built-in match) | — (unresolved tool need) |
| N4 | nested map-over + partial reduce; conditional per-gene override | `collection-flatten-after-fanout`, `conditional-transform-or-pass-through` |
| N5 | map-over with record-shaped multi-output per element | `sample_sheet:record` (see `galaxy-sample-sheet-collections.md`) |
| N6 | fan-in / tabular bridge from JSON | `tabular-concatenate-collection-to-table` (adapted; source is JSON not TSV) |
| N7 | extract-by-identifier | `collection-unbox-singleton` |

## 4. Unresolved Galaxy tool needs

1. **kmindex hit-map merge/union (N3).** No named existing Galaxy tool; today implemented as ad
   hoc BioBlend/Python (`harvest_collection()`, `process_kmindex_with_disassembler.py`). Needs
   either a bespoke Galaxy tool or a generic JSON-aggregation tool. Not resolved here.

2. **LexicMapStreamer (N5) — known source, wrapper authoring still pending, NOT "no existing
   implementation."** The user has confirmed the multi-HSP tiling/codon-QC/haplotype-collapsing
   logic behind `stream_lexicmap_msa.py` already exists as a real, working implementation in the
   private repository **`https://github.com/nekrut/disassembler`** — this is the actual code, not
   something to author from scratch by re-reading `stream_lexicmap_msa.py`'s prose description.
   What remains open: a Galaxy tool wrapper (XML + macros, or a Planemo-testable CLI wrapper) has
   not yet been authored against that repo's real CLI/API surface. `discover-shed-tool` should
   first check whether a Tool-Shed wrapper already exists for this repo before falling through to
   `author-galaxy-tool-wrapper`; if authoring is needed, it should target the repo's actual
   interface rather than reverse-engineering flags from the summary alone. This also affects the
   `--allow-frameshifts` default (item 3 below) and the exact shape of N5's 8 output artifacts —
   both should be confirmed by reading the real source, not left as unconfirmed prose.

3. **`--allow-frameshifts` default (N5 parameter).** Still unconfirmed, but no longer a blind
   unknown: this flag lives in the now-located `nekrut/disassembler` implementation and can be
   read directly once that repo is inspected during wrapper authoring, rather than inferred from
   `stream_lexicmap_msa.py`'s prose alone.

4. **kmindex/LexicMap DB- and index-selection surfacing (N2, N4).** The real Tool Shed tools'
   exact input schema for the 109 kmindex shards / 25 LexicMap indices has not been inspected
   (`summarize-galaxy-tool` has not run against the pinned versions). This is a property of the
   upstream IUC wrappers, not of LexicMapStreamer — the `nekrut/disassembler` discovery does not
   bear on it. Still open; see ledger.

5. **N6 JSON-to-tabular-row flattening.** No generic Galaxy tool flattens an arbitrary nested
   per-gene `.summary.json` into one manifest row; likely needs a small custom script/tool.

## 5. Placeholder (shape-changing) transformations

- N3: JSON collection (list, per-shard) → deduplicated accession list (single txt dataset) +
  implicit per-gene max-containment scores. Not a built-in Galaxy op.
- N4 reduce: `list:list` (gene × index) → `list` (gene only), collapsing the index axis while
  preserving gene `element_identifier`s and re-deriving any per-gene metadata the map-over step
  did not propagate (sample_sheet `column_definitions` are not carried onto tool outputs — see
  §1's `Gene query panel` row and the sample_sheet reference note).
- N5: LexicMap tabular hit table (per gene) → 8 heterogeneous output artifacts per gene
  (fasta ×3, tabular ×3, JSON ×1, fasta ×1) — the paper's core multi-HSP tiling/QC contribution;
  a record-shaped (`sample_sheet:record`) output is recommended over eight parallel collections.
- N6: nested JSON (per gene) → flat CSV row → concatenated manifest.
- N7: sample_sheet element (gene="E") → standalone dataset (unbox).

## 6. Confidence summary

- **High:** N1, N2, N4 (tool identity + named parameter sets), N5 (operation shape, per-gene
  multiplicity, 8-artifact output), N6 (intent).
- **Medium:** N3 (merge intent clear, mechanism ad hoc), N4 (conditional pick_value wiring,
  index-selection surfacing), N5 (`sample_sheet:record` recommendation, exact CLI-to-input
  mapping pending wrapper authoring), N6 (flattening mechanism), N7 (whether the am3 audit is an
  extracted element vs. a distinct 9th artifact).
- **Low / open:** whether `Reference genome (NC_001422.1)` is wired to any node at all (§7); the
  kmindex/LexicMap DB-and-index-selection Galaxy input surface (§4.4).

## 7. Open questions carried forward (not resolved by this brief)

1. Is the whole-genome `Reference genome (NC_001422.1)` input actually consumed by any Workflow A
   node? The source's own language ("canonical ΦX174 **gene** coordinate frame") points to the
   per-gene CDS FASTAs already inside `Gene query panel`, not the full circular genome. This brief
   does **not** invent a wiring the source doesn't support — flagged here rather than connected.
   If the template phase also finds no real consumer, consider dropping it from Workflow A's
   declared inputs (it may belong to Workflow B, where `NC_001422.1:2395-2919` is genuinely used).
2. Whether `Nominal-taxonomy BioProject classification table` should remain a declared Galaxy
   workflow input at all, given it is wired to nothing (see ledger
   `nominal-taxonomy-audit-not-modeled-as-step`) — an unconnected input is unusual for a runnable
   workflow; alternative is to drop it to documentation/README rather than a workflow input.
3. Whether N7's Gene E am3 audit JSON is literally the Gene E element of N5's `.summary.json`
   sample_sheet, or a ninth, separate artifact LexicMapStreamer also emits — resolvable once the
   `nekrut/disassembler`-backed wrapper's real output surface is known.
4. Exact Galaxy-native mechanism for kmindex/LexicMap DB- and index-selection (repeat vs.
   multi-select vs. per-element data input) — pending `summarize-galaxy-tool` against the pinned
   IUC changesets.

## 8. Open-requirements ledger cross-reference

See `open-requirements.ledger.yml` for full entries. This phase:

- **Resolved** `workflow-scope-boundary-unresolved` — user confirmed this run builds Workflow A
  only; Workflow B deferred as a nice-to-have second workflow; Workflows C/D out of scope.
- **Added** `lexicmapstreamer-implementation-provenance` (resolved) and
  `lexicmapstreamer-wrapper-authoring-pending` (open) to record the `nekrut/disassembler`
  discovery and the remaining wrapper-authoring work.
- **Updated note only (status unchanged, still open)** on `allow-frameshifts-default-unconfirmed`
  and `kmindex-lexicmap-index-selection-mechanism` to add the new provenance context.
- **Untouched:** `host-biome-platform-stratification-script-gap`,
  `nominal-taxonomy-audit-not-modeled-as-step`, `chronaeon-synthetic-metadata-phase0`,
  `chronaeon-hardcoded-statistics-phases`, `wei-2026-dms-dataset-external-sourcing`,
  `hyphaeon-toolshed-status-unverified` — all pertain to Workflows B/C/D, out of scope for this
  brief's wiring.

## 9. Foundry feedback

No feedback ledger entry was appended for this run. Rationale: the skill bundle's contract note,
ledger protocol note, sample-sheet note, and the collection/tabular/conditionals pattern MOCs
were sufficient to produce this brief — every difficulty encountered (ambiguous per-gene output
shape, ad hoc merge logic, an input that may not actually be wired) was a property of the source
project, not a defect or gap in the Foundry skill bundle itself.

• Galaxy Workflow Interface Design Brief
• 0. Scope decision made for this brief (see open-requirements entry `workflow-scope-boundary-unresolved`)
• 1. Workflow A — SRA Landscape & Spike-In Sieve (Stages A, B, C, D2)
• 1.1 Inputs
• 1.2 Outputs
• 2. Workflow B — Experimental-Evolution Trajectory Validation (Stage E)
• 2.1 Inputs
• 2.2 Outputs
• 3. Workflow C (candidate, lower confidence) — Diversity Reconstruction, VEP Benchmarking, Dual-Coding Calibration (Stages C′, F, G, H)
• 4. Workflow D (candidate, lowest confidence) — "ChronAeon" Multi-Scale Sieve (Stage I)
• 5. Collection-shape rationale
• 6. Labels and testability notes
• 7. Open questions carried to the open-requirements ledger
• 8. Foundry feedback

# Galaxy Workflow Interface Design Brief

Source: `freeform-summary.md` (ΦX174 planetary-scale retrospective DMS re-analysis pipeline).
This brief maps the free-form summary's Stages A-K into a Galaxy workflow interface: inputs,
outputs, labels, collection shapes, and checkpoints. It is a design handoff, not a gxformat2
skeleton — no wiring is committed here.

## 0. Scope decision made for this brief (see open-requirements entry `workflow-scope-boundary-unresolved`)

The source summary explicitly leaves open whether the target workflow is only the SRA-landscape /
spike-in sieve (Stages A-D2, the only content in `draft_manuscript.tex` today) or the full
multi-stage project (through HyphAeon/Fane synthesis, Stages A-K, documented in
HANDOVER.md/section drafts). This brief **covers the full A-K shape** so no source work is
silently dropped, but confidence drops sharply past Stage D2:

- **High confidence, likely one workflow ("SRA landscape & spike-in sieve"):** Stages A, B, C, D2.
- **Medium confidence, likely a second workflow ("experimental-evolution validation"):** Stage E.
- **Lower confidence, likely later/separate workflows pending tool-wrapper authoring:** Stages
  C′, F, G, H, I.
- **Out of Galaxy-workflow scope (non-computational or QA-only):** Stage D's manual metadata
  audit (modeled as reference data, not a step — see `nominal-taxonomy-audit-not-modeled-as-step`),
  Stage J (pure pandas literature cross-referencing), Stage K (figures / manuscript-style QA).

Downstream phases (data-flow, IWC-exemplar comparison, template authoring) should treat this as
a candidate decomposition into 2-4 connected workflows, not a mandate for one monolith.

## 1. Workflow A — SRA Landscape & Spike-In Sieve (Stages A, B, C, D2)

### 1.1 Inputs

| Label | Type | Collection shape | Description | Confidence |
|---|---|---|---|---|
| `Reference genome (NC_001422.1)` | `data` (fasta) | — | ΦX174 RefSeq genome, 5,386 nt circular ssDNA | High |
| `Gene query panel` | `data_collection` | **`sample_sheet`** — one FASTA per row, `element_identifier` = gene symbol (A,B,C,D,E,F,G,H,J,K) | Per-gene CDS FASTAs; also usable as a combined multi-FASTA for the bulk kmindex screen | High |
| `Gene query panel column_definitions` | (attached to above) | — | `align_min_match_pident` (float), `align_min_match_len` (int), `seed_min_prefix` (int), `min_qcov_per_genome` (float) — per-gene-tuned LexicMap sensitivity, needed because genes A/C/E/K use looser thresholds than the default | Medium — see rationale in §5 |
| `Logan kmindex DB selection` | `string` (select, multi-value) | — | Selection over the 109 hardcoded Logan k-mer shard names (`GENOMIC_BCT`, `VIRALRNA_*`, etc.) | Low — mechanism open, see ledger `kmindex-lexicmap-index-selection-mechanism` |
| `LexicMap index selection` | `string` (select, multi-value) | — | 25 domain indices, or the 5-index `TARGETED_PHAGE_INDICES` subset for per-gene runs | Low — same ledger entry |
| `kmindex parameters` | typed params | — | `zvalue=6`, `threshold=0.3`, `format=json`, `fast=false` | High |
| `LexicMap parameters` | typed params | — | `top_n_genomes=0`, `advanced_settings|all=true` | High |
| `Tiling/QC parameters` | typed params | — | `min_coverage=0.80`, `min_coverage_partial=0.50`, `min_pident=60.0`, `max_internal_stops=0`, `sample_cap=10`, `allow_frameshifts` (boolean, no confirmed default) | Medium — `allow_frameshifts` default open, see ledger |
| `Nominal-taxonomy BioProject classification table` | `data` (tabular, reference) | — | Static curation table (18 BioProjects → category) enumerated in Stage D; not computed by this workflow | Medium — modeled as reference data, see ledger `nominal-taxonomy-audit-not-modeled-as-step` |

### 1.2 Outputs

| Label | Type | Collection shape | Producer stage | Checkpoint? | Confidence |
|---|---|---|---|---|---|
| `kmindex containment hits (per shard)` | `data_collection` (JSON) | `list`, identifier = DB shard name | Stage B | No (intermediate, high volume) | High |
| `kmindex accession union` | `data` (txt) | — | Stage B (merge) | Yes — deterministic accession list | High |
| `LexicMap results (per gene)` | `data_collection` (tabular) | `sample_sheet` (mirrors query panel) | Stage C | No (raw hit table) | High |
| `Clean full-length CDS haplotypes (per gene)` | `data_collection` (fasta) | `sample_sheet` | Stage C tiling/QC | **Yes** — deterministic, drives Table 1 numbers | High |
| `Clean haplotype counts (per gene)` | `data_collection` (tabular) | `sample_sheet` | Stage C tiling/QC | **Yes** | High |
| `Flagged accessions + audit reasons (per gene)` | `data_collection` (fasta + tabular) | `sample_sheet` | Stage C tiling/QC | Yes | High |
| `Per-accession cohort ledger (per gene)` | `data_collection` (tabular) | `sample_sheet` | Stage C tiling/QC | No (audit trail, large) | High |
| `Per-gene ingestion summary` | `data_collection` (JSON) | `sample_sheet` | Stage C tiling/QC | **Yes** — source of Table 1 | High |
| `Ingestion manifest (all genes)` | `data` (csv) | — | Stage C aggregation | **Yes** — strong table checkpoint | High |
| `Gene E am3 quarantine audit` | `data` (JSON) | — | Stage D2 | **Yes** — small, deterministic, proves the paper's core spike-in finding | High |

## 2. Workflow B — Experimental-Evolution Trajectory Validation (Stage E)

### 2.1 Inputs

| Label | Type | Collection shape | Description | Confidence |
|---|---|---|---|---|
| `Reference genome (NC_001422.1)` | `data` (fasta) | — | shared with Workflow A | High |
| `Idaho 2024 paired reads` | `data_collection` | **`sample_sheet:paired`** — columns `timepoint` (int: 0/35/70 min), `replicate` (int) | 7 MiSeq PE runs, `SRR31059334`-`SRR31059340`, competitive-growth time series tracking Gene G | High |
| `Dickins & Nekrutenko 2009 single-end reads` | `data_collection` | **`sample_sheet`** — columns `lineage` (string, restrictions `[Ancestor,B,C]`), `sample_id` (string) | 10 single-end chemostat GAII samples | High |
| `Gene G coordinate window` | `string`/`int` params | — | `NC_001422.1:2395-2919`, used to restrict Idaho mpileup | High |
| `Historically important position list` | `string` (list) | — | Fixed Dickins positions (656, 1301, 1306, 1308, 1675, 3967, 4491, 5262) always retained regardless of frequency threshold | High |
| `Wei et al. 2026 DMS table (1nt/SNV level)` | `data` (tabular, external) | — | `genome_mut_ID_nt`-keyed fitness table; hard external dependency | Low — sourcing unresolved, see ledger `wei-2026-dms-dataset-external-sourcing` |

### 2.2 Outputs

| Label | Type | Collection shape | Checkpoint? | Confidence |
|---|---|---|---|---|
| `Sorted, indexed alignments (per sample)` | `data_collection` (bam) | `sample_sheet:paired` / `sample_sheet` | No (intermediate) | High |
| `Idaho 2024 trajectories vs DMS fitness` | `data` (csv) | — | **Yes** — carries Spearman/Pearson stats, N | High |
| `Dickins 2009 empirical trajectories` | `data` (csv) | — | **Yes** | High |

Both trajectory-caller scripts (`analyze_idaho_evolution.py`, `call_dickins_trajectories.py`) are
custom hand-rolled `samtools mpileup` string parsers with no off-the-shelf equivalent identified;
the source itself suggests `bcftools mpileup`/`lofreq`/`varscan` as a native-Galaxy substitute
path (freeform-summary §5) — flagged for the tool-discovery phase, not resolved here.

## 3. Workflow C (candidate, lower confidence) — Diversity Reconstruction, VEP Benchmarking, Dual-Coding Calibration (Stages C′, F, G, H)

Modeled at lower resolution because every non-kmindex/LexicMap tool here still needs a Galaxy
wrapper (per phase-1 context). Interface-level shape only:

| Label | Type | Collection shape | Description | Confidence |
|---|---|---|---|---|
| `Stratified diversity cohort accessions (per gene)` | `data_collection` (txt/json) | `sample_sheet` | 2,500 accessions/gene, 500/1000/1000 canonical/high-homology/divergent strata | Medium |
| `logan-walker cDBG traversal parameters` | typed params | — | `workers`, `fetch_workers`, `min_vaf=0.10`, `min_abund=5.0`, `prune_frac=0.01`, `min_cov=0.85`, `hops=5`, `max-size-mb=4096` | Medium |
| `Codon MSA / intra-host variant calls (per gene)` | `data_collection` (vcf-like + fasta) | `sample_sheet` | Stage C′ output | Medium — checkpoint candidate |
| `FastTree phylogenies (per gene)` | `data_collection` (newick) | `sample_sheet` | Stage C′ | Medium — checkpoint candidate |
| `EVcouplings MSAs (per gene, 350-taxa subsample)` | `data_collection` (fasta/aln) | `sample_sheet` | HyphAeon VEP input | Medium |
| `HyphAeon pathogenicity scores (per gene)` | `data_collection` (tabular) | `sample_sheet` | Stage G | **Yes** — checkpoint | Low-Medium (tool availability unverified, see ledger `hyphaeon-toolshed-status-unverified`) |
| `HyphAeon vs DMS correlation summary` | `data` (tabular) | — | Stage G | Yes | Low-Medium |
| `Overlap consequence classification (per gene-pair: DE, BA, KC, KA)` | `data_collection` (csv) | `list`, identifier = gene-pair code | Stage H | Yes | Low-Medium |
| `Shadow-effect calibration summary` | `data` (JSON) | — | Stage H | **Yes** — small, deterministic | Low-Medium |

## 4. Workflow D (candidate, lowest confidence) — "ChronAeon" Multi-Scale Sieve (Stage I)

`run_chronaeon_phix174_sieve.py` is an orchestration sketch, not a tool to port verbatim
(freeform-summary §7.7). Only Phases 1/3/4 are confirmed genuine re-executable CLI calls; Phases
0, 2, 6, 7, 8 are excluded from this brief as steps (see ledger entries
`chronaeon-synthetic-metadata-phase0`, `chronaeon-hardcoded-statistics-phases`).

| Label | Type | Collection shape | Description | Confidence |
|---|---|---|---|---|
| `Clean contigs + real collection-date metadata` | `data` (fasta) + `data` (csv) | — | Input to `hyphaeon autoclock`; **collection_date must be sourced from real SRA/BioSample metadata**, not the fabricated hash-derived placeholder in the source script | Low |
| `autoclock parameters` | typed params | — | `--manifold tn93`, `--max-depth 3`, `--min-leaf-size 20`, `--min-delta-aicc 15.0`, `--n-landmarks auto`, `--max-memory-mb 2048` | Medium |
| `autoclock summary` | `data` (JSON) | — | tmrca, rate, r2, optimal_k, delta_aicc, per-node classification | **Yes** — checkpoint | Low-Medium |
| `r0 parameters` | typed params | — | `--generation-time 0.0174`, `--generation-sd 0.005`, `--units days` | Medium |
| `r0 / phylodynamics summary` | `data` (JSON + csv) | — | epoch skyline Rt | **Yes** | Low-Medium |
| `meme parameters` | typed params | — | `--use-tn93`, `--attribute`, `--attribution-min-lrt 3.84` | Medium |
| `meme attributed sites` | `data` (JSON + csv) | — | episodic selection scan | **Yes** | Low-Medium |
| `ACAT combined p-values` | `data` (csv) | — | Cauchy combination over per-site p-values | **Yes** | Low-Medium |

## 5. Collection-shape rationale

- **`sample_sheet` over plain `list`/`list:paired`:** Several inputs pair one dataset per element
  with per-element scalar metadata that downstream steps consume as parameters, not just as
  grouping — per-gene LexicMap sensitivity thresholds (§1.1), Idaho timepoint/replicate, and
  Dickins lineage/sample_id. A `sample_sheet` (or `sample_sheet:paired`) carries that metadata as
  typed `column_definitions` instead of forcing parallel parameter inputs or baking values into
  filenames. This is a **medium-confidence design choice**: the source never describes Galaxy
  collection types explicitly (it is ad hoc BioBlend/local scripting today), so this is this
  brief's translation, not a source-stated requirement.
- Per the vendored sample_sheet reference, `sample_sheet` must be outermost and does not
  propagate `column_definitions` through a mapped tool automatically — the data-flow phase will
  need explicit re-attachment (e.g. via `__SAMPLE_SHEET_TO_TABULAR__` or rules-DSL) wherever
  per-gene tuning parameters must survive into Stage C's tiling/QC step.
- Gene-pair overlap outputs (`DE`, `BA`, `KC`, `KA`) are modeled as a plain `list` (not
  `sample_sheet`) since they carry no per-element scalar metadata beyond the identifier itself.

## 6. Labels and testability notes

- Labels above are chosen as stable domain names (e.g. `Gene E am3 quarantine audit`, `Idaho 2024
  trajectories vs DMS fitness`) rather than tool-step defaults, per the testability-design
  guidance — these are the identifiers a later `-tests.yml` would address.
- Checkpoints favor small deterministic JSON/CSV summaries (the per-gene `.summary.json` files,
  the am3 audit JSON, the trajectory CSVs, the shadow-effect calibration JSON) over the large
  audit-trail collections (cohort ledgers, raw hit tables) — those remain workflow outputs but are
  not proposed as primary test-assertion targets given their size/volume.
- `sample_sheet`-shaped collection outputs should carry gene symbol / sample_id as
  `element_identifier` throughout, so element-level test assertions (`element_tests:`) can target
  them directly.

## 7. Open questions carried to the open-requirements ledger

See `open-requirements.ledger.yml` for the full entries (9 total, all `status: open`, none
`blocking` — these are design-tier gaps/drops, not computability gaps in a draft topology):

1. `workflow-scope-boundary-unresolved` — Stages A-D2 only, vs full A-K.
2. `kmindex-lexicmap-index-selection-mechanism` — how 109/25 Logan index names surface as tool inputs.
3. `allow-frameshifts-default-unconfirmed` — QC flag default.
4. `host-biome-platform-stratification-script-gap` — no script found; dropped, not invented.
5. `nominal-taxonomy-audit-not-modeled-as-step` — modeled as reference data, not a step.
6. `chronaeon-synthetic-metadata-phase0` — fabricated dates excluded, real dates needed.
7. `chronaeon-hardcoded-statistics-phases` — Phases 2/6/7/8 excluded as non-reproducible.
8. `wei-2026-dms-dataset-external-sourcing` — external DMS ground-truth table not in repo.
9. `hyphaeon-toolshed-status-unverified` — HyphAeon/hyphaeon_overlaps Tool Shed status unknown.

## 8. Foundry feedback

No feedback ledger entry was appended for this run. Rationale: the skill's `_feedback.md`
protocol and the packaged reference notes (`open-requirements-ledger.md`,
`galaxy-sample-sheet-collections.md`, `galaxy-workflow-testability-design.md`) were sufficient to
make every interface decision above; nothing encountered pointed at a defect, contradiction, or
uncovered case in the Foundry skill bundle itself. The difficulties in this run (unresolved
project scope, an unwrapped multi-stage orchestration script, an unverified external tool's
Tool-Shed status, a missing external dataset) are properties of this project's source material,
not of the Foundry asset, so per the ledger's own scope rule they were recorded in
`open-requirements.ledger.yml` instead.

• Free-Form Source Summary: Planetary-Scale Retrospective DMS of Bacteriophage ΦX174
• 0. Scope note (read this first)
• 1. High-level pipeline shape
• 2. Stage-by-stage detail
• Stage A — Reference genome and per-gene CDS query panel
• Stage B — Petabase k-mer containment screen (kmindex, real Galaxy tool)
• Stage C — Track 1: LexicMap streaming search (real Galaxy tool) + custom tiling/QC client
• Stage C′ — Track 2: Disassembler / `logan-walker` cDBG traversal (real external tool, Rust)
• Stage D — Nominal-taxonomy SRA metadata audit
• Stage D2 — The Sanger `am3` dual-coding molecular fingerprint (core finding, not a tool stage)
• Stage E — Reference experimental-evolution datasets (real tools: bowtie2 + samtools)
• Stage F — Wei, Li & Lehner (2026) whole-genome DMS ground truth (external dataset, NOT in repo)
• Stage G — HyphAeon zero-shot variant-effect prediction (real installed tool)
• Stage H — Dual-coding / overlapping-frame consequence classification (custom package, real code)
• Stage I — "ChronAeon Multi-Scale SRA Sieve" (`run_chronaeon_phix174_sieve.py`)
• Stage J — Fane-mechanics and Bull/Wichman literature cross-referencing (pure pandas, no external tool)
• Stage K — Figure generation and manuscript QA (project tooling, not science tools)
• 3. Sample / reference data catalog (for test-data resolution phases)
• 4. `results/` directory shape (already-computed intermediate/output tables — useful as
• 5. Tool inventory — real/installable vs. custom/internal
• 6. Key literature / reference data anchors (for citation/README generation downstream)
• 7. Assumptions and open gaps (carried forward per skill protocol)
• 8. Foundry feedback

# Free-Form Source Summary: Planetary-Scale Retrospective DMS of Bacteriophage ΦX174

## 0. Scope note (read this first)

The primary source is `/Users/scottcain/git/dms/paper/draft_manuscript.tex` ("The Petabase
Evolutionary Landscape of Bacteriophage ΦX174: Deconvoluting Laboratory Spike-In Clouds,
Dual-Coding Constraints, and Uncultivated Microviral Diversity", Nekrutenko et al., draft,
Sept 2026). **As it stands today, this .tex file contains only the Introduction and the first
Results section** (SRA landscape certification / spike-in sieve, ~6 pages). It does not yet
contain the later results sections (real-time purging kinetics, HyphAeon VEP benchmarking,
dual-coding shadow effect, Fane morphogenesis) that are drafted in `sections/02..07_*.md` and in
the sibling manuscript `phiX174_planetary_dms.tex` (explicitly out of scope per instructions).

Because the task asked for the *computational methods* underlying the whole project (this
summary feeds a Galaxy-workflow-reconstruction pipeline, not a manuscript-text pipeline), this
document covers the **full computational pipeline implied by draft_manuscript.tex plus the
supplementary project material** (HANDOVER.md, PROJECT_SUMMARY.md, README.md, the ChronAeon/
HyphAeon reanalysis report, the per-section drafts, and the actual Python/shell scripts at the
repo root). Facts that come only from the current draft_manuscript.tex are marked **[PAPER]**;
facts that come from the broader project context are marked **[CONTEXT]**. Downstream phases
should treat **[PAPER]** facts as the authoritative scope of "the paper" and **[CONTEXT]** facts
as enrichment / rationale for why each pipeline stage exists.

The manuscript's actual object of study is not a single wet-lab experiment: it is a
**retrospective computational re-analysis of public sequence archives** (the NCBI SRA / Logan
petabase index) for the historically important model virus ΦX174, cross-validated against a
2026 whole-genome saturation-mutagenesis (deep mutational scanning, DMS) dataset and several
classical experimental-evolution datasets. The reproducible "workflow" is therefore a data
mining + curation + statistics pipeline, not a wet-lab protocol.

## 1. High-level pipeline shape

```
[Stage A] Reference & query prep (genome, 10 CDS FASTAs)
        |
[Stage B] Petabase k-mer containment screen: kmindex (Galaxy tool) across 109 Logan k-mer DBs
        |
[Stage C] Track 1 — LexicMap sequence-to-graph streaming search (Galaxy tool) across Logan indices
        |         -> custom streaming client (stream_lexicmap_msa.py) does multi-HSP coordinate
        |            tiling, codon QC, haplotype collapsing -> clean/flagged FASTA + TSV + JSON
        |
[Stage C'] Track 2 — Disassembler/logan-walker cDBG traversal (Rust binary) on stratified
        |            diversity cohorts -> intra-host SNVs/haplotypes (VCF-like / codon MSA)
        |
[Stage D] Nominal-taxonomy SRA metadata audit (manual/registry-based BioProject certification)
        |
[Stage E] Reference experimental-evolution datasets ingested independently:
        |   - Idaho 2024 SRA time series (download -> bowtie2 align -> samtools mpileup -> custom
        |     Python trajectory caller) -> selection-rate vs DMS-fitness correlation
        |   - Dickins & Nekrutenko 2009 chemostat GAII reads (Galaxy library download -> bowtie2
        |     -> samtools mpileup -> custom trajectory caller)
        |
[Stage F] Wei/Li/Lehner 2026 whole-genome DMS dataset (external ground truth, NOT bundled in
        |   this repo — see Gaps) joined against every other stage by mutation ID
        |
[Stage G] HyphAeon (real installed CLI + Python package) zero-shot variant-effect prediction,
        |   benchmarked against GEMME/ProteoCast/ESCOTT/RSALOR/DDMut/ESM-2/ESM3/Tranception
        |
[Stage H] hyphaeon_overlaps (custom package) dual-coding consequence classification + shadow-
        |   effect calibration on overlapping ORFs (D/E, B/A, K/C, K/A)
        |
[Stage I] "ChronAeon" multi-scale sieve = a fixed sequence of HyphAeon CLI subcommands
        |   (autoclock -> r0 -> meme -> filter) plus closed-form stats (ACAT/Cauchy combination)
        |   run over the curated sequence cohorts, in run_chronaeon_phix174_sieve.py
        |
[Stage J] Fane-mechanics / Bull-Wichman literature cross-referencing (pandas joins against DMS
        |   table + curated bibliography JSON, no external tool)
        |
[Stage K] Figure generation (matplotlib/seaborn) + manuscript style linting (project-internal,
            not a science tool)
```

## 2. Stage-by-stage detail

### Stage A — Reference genome and per-gene CDS query panel
- **[PAPER/CONTEXT]** Reference genome: NCBI RefSeq `NC_001422.1` (ΦX174, 5,386 nt circular
  ssDNA). File on disk: `ref/NC_001422.1.fasta` (+ `.fai`).
- 10 canonical ORFs treated as the query/target set: **A, B, C, D, E, F, G, H, J, K** (an 11th,
  A*, is an internal in-frame product of A, not separately queried in most scripts although
  listed in some contexts). Per-gene lengths (bp/aa) and roles, from the paper's Table 1:
  - J 114/38 (DNA core packaging), K 168/56 (overlap regulator, K overlaps A and C), C 258/86
    (DNA synthesis switch), E 273/91 (lysis, MraY inhibitor), B 360/120 (internal scaffold), D
    456/152 (external scaffold, 240-mer cage), G 525/175 (major spike pentamer), H 984/328
    (pilot/DNA-ejection protein), F 1,281/427 (major capsid, T=1 60-mer), A 1,539/513
    (rolling-circle replicase). Total coding length 5,958 bp (exceeds genome length 5,386 nt
    because of overlapping frames).
- Individual gene FASTAs at `cds/A.fasta` … `cds/K.fasta`, `cds/Astar.fasta`, plus a combined
  query panel `cds/phix174_all_genes_panel.fasta` used as the multi-FASTA input to the Galaxy
  kmindex tool.
- Protein reference set: `ref/phix174_proteins.faa` / `.fasta`.
- Historical/auxiliary genomics pulls (not central to the pipeline but present):
  `data/ncbi_virus_phix174_annotated.tsv`, `ncbi_virus_phix174_raw.json`,
  `ncbi_virus_complete_genomes.fasta`, `ncbi_virus_phix174.gb` — an NCBI Virus resource pull used
  for the "185 complete GenBank records / 91.8% derive from lab experimental evolution"
  certification mentioned in HANDOVER.md **[CONTEXT]**.

### Stage B — Petabase k-mer containment screen (kmindex, real Galaxy tool)
- **[PAPER, "high-throughput k-mer containment indexing (kmindex)"]** Tool: **kmindex**, invoked
  as an actual installed Galaxy Tool Shed wrapper, not a hypothetical name:
  `toolshed.g2.bx.psu.edu/repos/iuc/kmindex/kmindex_query/0.6.1+galaxy4`, run via BioBlend
  (`submit_kmindex_all_genes.py`).
- Databases queried: **109 Logan SRA k-mer index shards**, organized into a fixed taxonomic/
  library-type grid, e.g. `GENOMIC_BCT`, `GENOMIC_HUMAN`, `GENOMIC_VRL`, `METAGENOMIC_ENV`,
  `METATRANSCRIPTOMIC_*`, `SYNTHETIC_*`, `VIRALRNA_*`, `TRANSCRIPTOMICSINGLECELL_*`, etc. (full
  list of 109 DB names hardcoded in `ALL_KMINDEX_DBS`).
- Concrete kmindex parameters used: `zvalue=6`, `threshold=0.3`, `format=json`, `fast=False`,
  `verbose=error`. Query input: the combined multi-FASTA panel `cds/phix174_all_genes_panel.fasta`
  uploaded as a Galaxy HDA.
- Output: per-shard JSON hit maps (accession -> containment score), tracked in
  `kmindex_runs.json`; harvested/merged later (Stage C') into `{gene}.scores.json` and
  `UNION.accessions.txt` (2,114,904 unique accessions cited in the paper).
- Job orchestration is via the Galaxy REST API / BioBlend against `https://usegalaxy.org`, using
  an API key file at `/tmp/gxy.txt` — i.e., **this stage already runs on Galaxy today** via
  ad-hoc scripting rather than a formal workflow; a major goal of this Foundry pipeline is
  presumably to formalize exactly this into a `.ga`/gxformat2 workflow.

### Stage C — Track 1: LexicMap streaming search (real Galaxy tool) + custom tiling/QC client
- **[PAPER]** Tool: **LexicMap**, "linear sequence-to-graph streaming aligner", also a real
  installed Galaxy tool: `toolshed.g2.bx.psu.edu/repos/iuc/lexicmap/lexicmap_search/0.9.0+galaxy1`
  (`pull_logan_alignments.py`). Indices queried are Logan-derived per-domain LexicMap indices:
  `BacteriaGenomic`, `BacteriaMetagenomic`, `HumanGenomic`, `HumanMetagenomic`, `MouseGenomic`,
  `Viral`, `Synthetic`, `Singlecell`, etc. (25 total; a narrower `TARGETED_PHAGE_INDICES` subset
  of 5 — `Viral`, `BacteriaMetagenomic`, `HumanMetagenomic`, `OtherMetagenomic`, `Synthetic` — is
  used for per-gene targeted runs). Parameters used: `top_n_genomes=0`, `advanced_settings|all=True`
  (i.e. return all hits, no top-N truncation). For the smaller/harder-to-recover genes (`submit_remaining_logan.py`,
  covering genes A/C/E/K), per-gene-tuned search sensitivity parameters are used instead of
  defaults: `align_min_match_pident=60.0`, `align_min_match_len` in the 35–50 range,
  `seed_min_prefix` in the 15–17 range, `min_qcov_per_genome=30.0` — i.e. short/divergent genes
  need looser seed/match thresholds to be recovered at all.
- Per-gene submission/monitor/download cycle implemented in `pull_logan_alignments.py`,
  `submit_all_structural_logan.py`, `submit_remaining_logan.py`, `process_and_benchmark_logan.py`
  — all via BioBlend against usegalaxy.org, tracked in JSON state files
  (`logan_structural_runs.json`, `logan_remaining_runs.json`, `logan_job_{gene}.json`).
- Output of the Galaxy tool: a results TSV per gene (`logan_results/{gene}.tsv` or
  `{gene}.results.tsv`) — a BLAST/LexicMap-style tabular hit table with columns including
  `sgenome` (SRA run accession), percent identity (`pident`), and HSP coordinates.
- **[PAPER]** The custom rescue step (novel algorithmic contribution, `stream_lexicmap_msa.py`,
  resolves "GitHub Issue #1" on `nekrut/dms`) streams this TSV (from a local file, stdin, or
  directly over HTTP from a Galaxy dataset `/api/datasets/{id}/display` endpoint using an API key
  at `/tmp/gxy.txt`), groups hits by SRA accession, and:
  1. Merges multiple non-overlapping HSPs per accession onto the canonical ΦX174 gene coordinate
     frame (greedy tiling, ties broken by higher `pident`) — "multi-HSP coordinate tiling".
  2. Translates using the standard/bacterial genetic code (NCBI table 11, hardcoded codon table
     in-script) and flags premature internal stop codons (excluding the natural terminal stop).
  3. QC gate, exposed as CLI flags on `stream_lexicmap_msa.py`: `--min-coverage 0.80` (full-length
     clean cut), `--min-coverage-partial 0.50` (partial/expanded cut), `--min-pident 60.0`,
     `--max-internal-stops 0`, `--sample-cap 10` (per-cohort ledger sampling), and an
     `--allow-frameshifts` flag whose default/production setting could not be confirmed from prose
     alone — treat frameshift handling as a parameter to confirm before Galaxy-wrapping this step,
     not an assumed "always reject" behavior.
  4. Partitions each accession into **Clean** vs **Flagged** cohorts and collapses Clean
     sequences into non-redundant haplotypes (weighted by observation count).
- Declared outputs per gene (prefix `{gene}`): `.clean.msa.fasta`, `.clean.haplotypes.tsv`,
  `.clean.accessions.fasta`, `.clean_expanded.accessions.fasta` (partial ≥50% cov), 
  `.flagged.accessions.fasta`, `.flagged.tsv` (audit reasons), `.cohort_ledger.tsv` (per-accession
  status for every evaluated accession), `.summary.json` (quantile distributions, regime
  diagnostics, multi-HSP recovery counts). These per-gene summary JSONs
  (`msa/phiX174_{gene}.summary.json`) are the direct source of the paper's Table 1 numbers and
  are aggregated into `results/00_ingestion/lexicmap_9genes_ingestion_manifest.csv`.
- **[PAPER]** Quantitative outcome (Table 1 in draft_manuscript.tex): 26,836,801 streamed rows /
  24,378,503 evaluated accession queries across the 10 genes; 1,060,472 multi-HSP rescues;
  17,921,411 clean full-length CDS; 316,135 unique clean haplotypes; 2,218,490 flagged internal
  stops (2,215,172 of which are in Gene E alone — the `am3` spike-in, see Stage D2 below).

### Stage C′ — Track 2: Disassembler / `logan-walker` cDBG traversal (real external tool, Rust)
- **[CONTEXT]** A second, independent analytical track described in PROJECT_SUMMARY.md /
  HANDOVER.md and implemented in `process_kmindex_with_disassembler.py`. Not mentioned in the
  current draft_manuscript.tex text, but represents real, runnable code.
- Real external binary: `logan-walker`, a Rust program built at
  `~/git/disassembler/walker/target/release/logan-walker` (project `nekrut/disassembler`,
  upstream PRs #3 and #4 referenced in README.md). Companion Python script:
  `build_walker_codon_msa.py` (same repo, `python/` dir) and `build_ref_guided.py`
  (`nekrut/logan-hypyaeon`, referenced from `pull_logan_alignments.py`).
- Purpose: traverse compacted de Bruijn graphs (cDBGs) directly from Logan's public S3 bucket
  (`s3://logan-pub`) per SRA accession, recovering intra-host alleles/SNVs and structural bubbles
  without reference-alignment bias (contrasts with Track 1's linear/reference-guided approach).
- Concrete parameters (defaults seen in `run_disassembler_for_gene(...)`,
  `process_kmindex_with_disassembler.py`): `workers=8`, `fetch_workers=6`, `min_vaf=0.10`,
  `min_abund=5.0`, `min_abund_frac=0.0`, `prune_frac=0.01` ("two-pass depth pruning": unitigs
  below 1% of gene depth pruned before realignment), `min_cov=0.85`; elsewhere (PROJECT_SUMMARY.md)
  cited as `--hops 5`, `--max-size-mb 4096`, `--emit-vcf`, `--no-emit-alts` (biallelic bubble
  phasing without chimeric alt-haplotype emission), followed by codon-level phylogeny via
  **FastTree** (real tool, GTR nucleotide model; binary at
  `~/miniconda3/envs/logan/bin/fasttree`, invoked from `process_and_benchmark_logan.py`).
- Input: harvested per-gene candidate accession lists from Stage B's merged kmindex shard JSONs
  (`harvest_collection()` in `process_kmindex_with_disassembler.py` walks a Galaxy dataset
  collection of ~2,869 shard JSON files in parallel, 32 worker threads, via the Galaxy history
  contents API). Output: `{gene}.accessions.txt`, `{gene}.scores.json`, `UNION.accessions.txt`
  (per-gene max containment scores; feeds Stage below).
- Because full reconstruction of all 2.1M accessions would require >72 TB of S3 downloads
  (PROJECT_SUMMARY.md), a **stratified diversity cohort of 2,500 accessions/gene** is drawn
  instead (`create_targeted_diversity_cohorts.py`): 500 "canonical" (containment ≥0.99, control
  spike-in baselines), 1,000 "high-homology" (0.80–0.99), 1,000 "divergent" (0.50–0.80), evenly
  sampled across each score bracket (not just top-N) to avoid clustering. Outputs:
  `kmindex_candidates/{gene}.diversity_2500.txt/.json`, `diversity_2500_summary.json`.

### Stage D — Nominal-taxonomy SRA metadata audit
- **[PAPER]** Not a computational tool per se but a manual/registry-based certification: all SRA
  runs filed under NCBI TaxID `10847` ("Escherichia phage phiX174") and `2886930`
  ("Sinsheimervirus phiX174") were enumerated (136 runs / 18 BioProjects) and each BioProject was
  manually classified into one of: Experimental Evolution (7 runs, `PRJNA1174868`/`SRP539859`,
  the Idaho 2024 study — the *only* genuine evolution study), Platform Benchmarking (36 runs;
  `SRP042938`, `SRP008975`, `ERP002532`, `ERP000074`, `SRP001260`), Paleogenomics Control (35 runs;
  `ERP001254`, MPI-EVA ancient-DNA calibration), Metagenomics/Cross-talk (29 runs; `SRP347090`,
  `SRP653402`, `ERP001281`, `SRP066453`), Method Development (18 runs; `SRP059208`, `SRP082602`
  — Cir-Seq/Droplet-Seq/O2N-Seq benchmarks), Misfiled Spike-Ins (6 runs; `SRP009433`/`SRR009433`,
  `ERP000177` GEUVADIS, `SRP003315` Drosophila, `SRP042938`), Synthetic Standards (1 run;
  `SRP363953`, pREF), General QC (4 runs; `DRP008496`, ribonucleoside misincorporation).
- Result: only 5.1% (7/136) of nominally-tagged runs are genuine evolution experiments; this
  motivates bypassing nominal taxonomy entirely and mining the full Logan index by sequence
  content instead (Stages B/C above).
- Host/biome/platform metadata stratification of the 2,114,904-accession Logan hit set (human
  clinical 14.0%, human microbiome 8.7%, model animals/livestock 20.7%, environmental
  metagenomes 18.1%, crops 8.0%, pathogen surveillance 6.4%, bacterial isolates 3.3%; platforms:
  100% Illumina — MiSeq 28.4%, HiSeq family ~48%, NovaSeq 12%) appears to be produced by
  cross-referencing SRA run metadata (BioSample/BioProject attributes) against the accession
  union list; no single script for this was identified among the root-level `.py` files — likely
  done via NCBI Entrez/SRA metadata API calls not captured in a standalone script, or via manual
  curation. **This is a gap for downstream reproducibility (see Section 5).**

### Stage D2 — The Sanger `am3` dual-coding molecular fingerprint (core finding, not a tool stage)
- **[PAPER]** A specific diagnostic mutation is used to prove that most global ΦX174 SRA hits are
  the Illumina PhiX Control v3 library (derived from the 1977 Sanger `am3` amber mutant), not wild
  biology: genome position **nt 587, G→A**. In Gene E (+1 frame, nt 568–843) this converts codon 7
  TGG→TAG (`gpE_W7*`, premature amber stop, abolishes MraY inhibition / lysis). In Gene D (0
  frame, nt 390–848) the same nucleotide is the third position of codon 66, GTC→GTA
  (`Val66Val`, 100% synonymous, procapsid scaffold unaffected). 2,215,172 / 2,392,457 (92.59%)
  of evaluated Gene E accessions carry this exact substitution; quarantining them recovers
  52,862 bona-fide wild-type Gene E sequences. This detection logic lives inside the QC step of
  `stream_lexicmap_msa.py` / is summarized in `results/02_layer1_quasispecies/
  gene_e_am3_quarantine_audit.json`.
- Downstream statistical consequence (`P(error) = 1 - e^{-Lε}` with Illumina substitution rate
  ε≈10⁻³/nt): long genes (F 1,281 bp, A 1,539 bp) accumulate singleton haplotypes at 89.9%/84.3%
  of all unique haplotypes; stratified identity curation (>99% identity = error cloud vs
  50–98% = genuine divergent microvirus) is required before any VEP benchmarking (Stage G).

### Stage E — Reference experimental-evolution datasets (real tools: bowtie2 + samtools)
- **Idaho 2024** (`PRJNA1174868` / `SRP539859`): 7 Illumina MiSeq paired-end runs
  `SRR31059334`–`SRR31059340`, competitive-growth chemostat time series at t=0, 35, 70 min,
  tracking Gene G. Download: `download_idaho_sra.py` (parallel `urllib` fetch against a manifest
  `phix174_sra_evolution_manifest.tsv` with per-file MD5 checksums, 4 worker threads) into
  `data/sra_evolution/idaho2024/`. Alignment: `align_idaho_sra.sh` —
  **bowtie2** (`bowtie2 -x ref/NC_001422.1 -1 R1 -2 R2 -p 4`) piped to **samtools sort**
  (`samtools sort -o {run}.sorted.bam -`) then **samtools index**. Both binaries are pinned to a
  specific conda env (`~/miniconda3/envs/mrsa_align/bin/{bowtie2,samtools}`).
- Trajectory calling (`analyze_idaho_evolution.py`): **samtools mpileup** restricted to Gene G
  coordinates (`-r NC_001422.1:2395-2919 -f ref/NC_001422.1.fasta -d 500000 -A -q 0`) across all
  7 BAMs simultaneously; a hand-rolled pileup-string parser (regex-based, strips read-start/end
  markers and indel length prefixes) tallies per-sample allele counts; for every non-reference
  allele at every position it computes codon context (via Biopython `CodonTable.unambiguous_dna_by_id[1]`,
  standard genetic code) to classify Synonymous/Missense/Nonsense, computes `f0`, mean `f35`
  (avg of 3 replicates), mean `f70` (avg of 3 replicates), a linear slope
  `(f70-f0)/70` and a log-ratio selection rate `s = ln((f70+ε)/(f0+ε))/70` with `ε=1e-5`, joins
  against the Wei 2026 DMS table by `genome_mut_ID_nt`, and reports Spearman/Pearson correlation
  between `s` and DMS fitness `w` (filtering to `freq_t0 > 0.0005` and non-null DMS fitness).
  Output: `idaho2024_trajectories_vs_dms.csv`. Reported result: Spearman ρ=0.3549 (P=5.9e-44),
  Pearson r=0.3632 (P=4.3e-46), N=1,439 SNVs; lethal-allele mean s=-0.0195 min⁻¹.
- **Dickins & Nekrutenko 2009** (`GBE` 1:294-307, chemostat GAII data): 10 samples (Ancestor A1–A4,
  Lineage B: B1,B3,B4, Lineage C: C1,C3,C4). Download via **BioBlend** from a specific existing
  Galaxy data-library folder (`download_dickins_galaxy.py`, folder ID `F175d7d5ddeeb1d43` on
  `https://usegalaxy.org`, imported into a history then downloaded per-HDA and gzip-compressed).
  Alignment: `align_dickins.sh` — same bowtie2 (single-end `-U`) + samtools sort/index pattern as
  Idaho, against the same `NC_001422.1` reference.
  Trajectory calling: `call_dickins_trajectories.py` — whole-genome **samtools mpileup**
  (`-d 500000 -A -q 0`, no coordinate restriction) across all 10 BAMs, same custom pileup parser,
  keeps any position with max allele frequency ≥0.5% across samples or matching a fixed list of
  historically important positions (656, 1301, 1306, 1308, 1675, 3967, 4491, 5262), joins against
  the Wei 2026 DMS table. Output: `dickins2009_empirical_trajectories.csv`. Key validated
  mutations: `C656A` (`gpE_S30*`, rises 0%→0.76%→1.58% in Lineage B; DMS fitness w=0.5829 — the
  "17-year lysis paradox"), `A1301G` (`gpF_T101A`, sweeps to 78.5%/32.7% in Lineages B/C; w=1.039),
  `G319T`/`A323G` (`gpC_V63F`/`gpC_D64G`, reach 11.0%/8.4%).

### Stage F — Wei, Li & Lehner (2026) whole-genome DMS ground truth (external dataset, NOT in repo)
- **[PAPER via citation; CONTEXT for file format]** bioRxiv doi:10.64898/2026.07.25.740675,
  "Complete Mutagenesis of the Genome and Proteome of ΦX174." N=39,335 amino-acid variants /
  16,098 SNVs across all 11 genes, competitive-growth relative fitness `w` for every accessible
  substitution.
- Every downstream script expects this at a **fixed relative path that does not exist in this
  checked-out repo**: `phix174_WGM/data/phix_all_dms100_sub_libraries_normalized_unique_stat_1nt.tsv.zip`
  (the 1nt/SNV-level table, columns include `genome_mut_ID_nt`, `phix_aa_mut_genes`,
  `phix_aa_mut_ID`, `phix_aa_mut_class`, `phix_aa_mut_class_sum`, `fitness_final`,
  `mutation_category_FDR_lethal_1nt`, `genome_nt_pos_overlap_orf`, per-gene per-position columns
  like `gene_sub_aa_mut_gene`/`_pos`/`_class`) and
  `phix174_WGM/data/evaluate_VEPs/phix_proteins_dms_mut_modelling_merged.tsv.zip` (protein-level
  table merged with precomputed VEP scores). **This external dataset is a hard input dependency
  for Stages E/G/H/J and must be sourced (from the Wei et al. bioRxiv supplement / GitHub) for
  any reproduction — see Gaps, Section 5.**

### Stage G — HyphAeon zero-shot variant-effect prediction (real installed tool)
- **[CONTEXT, confirmed by code]** HyphAeon ("Phylogenetic Axial Transformer") is **a real,
  installed multi-purpose bioinformatics package/CLI**, not merely a name used in prose. Evidence:
  `from hyphaeon.inference import load_model, get_device` and
  `from hyphaeon.disease import predict_disease_pathogenicity` (Python API, used in
  `benchmark_hyphaeon_phix174.py` and `process_and_benchmark_logan.py`, running on `torch`/GPU —
  "12.4 s on a single NVIDIA A100"); AND a separate CLI binary
  `~/miniconda3/bin/hyphaeon` invoked via `subprocess` with real subcommands
  `autoclock`, `r0`, `meme`, `filter` (see Stage I). Upstream project: `github.com/veg/hyphaeon`
  (per section drafts) — this is the Kosakovsky Pond lab's phylogenetics toolkit, i.e. plausibly
  installable as an actual Galaxy/conda tool, not project-internal glue code.
- Python-API usage (`predict_disease_pathogenicity`): inputs an MSA file path (from
  `msas_sub/{gene}_b*` — subsampled 350-taxa EVcouplings-format MSAs) and a list of mutant codes
  (parsed via regex `gp[A-Z]_([A-Z]\d+[A-Z])` from DMS IDs), scores pathogenicity per mutation,
  correlated (Spearman) against DMS `fitness_aa_mut`.
- Benchmarked against pre-computed columns already present in the input TSV for other VEP tools
  (not run by this repo, just compared): **GEMME**, **ProteoCast**, **ESCOTT**, **RSALOR**,
  **DDMut**, **ESM_IF1**, **MIF_ST**, plus separately **ESM-2 (15B)**, **ESM3 Open (1.4B)**,
  **Tranception-L** (all real published external tools/models, values taken from the merged DMS
  table, not re-run here).
- Reported results (Table 3 in the fuller manuscript draft, not yet in draft_manuscript.tex):
  HyphAeon proteome mean ρ=+0.149 (median +0.131) vs ESM-2 15B mean ρ=+0.013; per-gene range
  ρ=+0.071 (B) to +0.274 (D); stratified-curation gain 3.2× on Gene F (ρ +0.047→+0.154) and rescue
  of Gene G from negative correlation (ρ -0.011→+0.118, P=7.7e-6).

### Stage H — Dual-coding / overlapping-frame consequence classification (custom package, real code)
- **[CONTEXT, confirmed by code]** Package `hyphaeon_overlaps` (repo `nekrut/hyphaeon-overlaps-cdx`
  / `nekrut/hyphaeon-overlaps`), imported as
  `from hyphaeon_overlaps.consequences import consequence, joint_class` in
  `calibrate_hyphaeon_shadow_effect.py`. This is genuinely a separate installable Python package
  (added to `sys.path` from a sibling checkout `~/git/hyphaeon-overlaps-cdx`), i.e. NOT part of
  core HyphAeon, but a purpose-built companion library for classifying joint consequence codes
  (SS/SN/NS/NN/Stop-gain/Stop-loss) of a single nucleotide change across two overlapping reading
  frames.
- Analysis: filters DMS table to `genome_nt_pos_overlap_orf == "Overlapping Coding"`, computes
  joint class per row, groups by gene-pair (`DE`, `BA`, `KC`, `KA`) and joint class, runs
  `scipy.stats.mannwhitneyu` / `ks_2samp` / `ranksums` between asymmetric categories (NS vs SN)
  to test directional selection asymmetry, and defines the "empirical shadow penalty"
  `Δw_shadow = 1.0 - median(w_SN)`. Outputs: `de_overlap_dms_summary.csv`,
  `ba_overlap_dms_summary.csv`, `hyphaeon_overlap_calibration_results.csv`,
  `results/07_overlapping_frames/shadow_effect_calibration.json`.
- Key reported numbers: Gene D/E Mann-Whitney P=1.15e-42 (asymmetry ratio 4.01), Gene B/A
  P=5.96e-7 (ratio 1.50); shadow penalty severe in replicase gpA (Δw=0.359), near-zero in lysis
  gpE (Δw=0.051).

### Stage I — "ChronAeon Multi-Scale SRA Sieve" (`run_chronaeon_phix174_sieve.py`)
- **[CONTEXT]** This is the most structurally important script for a Galaxy workflow translation:
  it is a literal multi-phase pipeline script, explicitly implementing spec doc
  `SPEC-CHRONAEON-HYPHAEON-SRA-03` referenced in `CHRONAEON_HYPHAEON_PHIX174_REANALYSIS_REPORT.md`.
  "ChronAeon" itself does not appear to be a separate installed tool — it is this orchestration
  script's name for a *sequence of real HyphAeon CLI subcommands* plus custom pandas/numpy glue:
  - **Phase 0 (ingestion)**: aggregate per-gene `msa/phiX174_{gene}.summary.json` files (produced
    by Stage C) into `results/00_ingestion/lexicmap_9genes_ingestion_manifest.csv`; build a
    representative cross-scale FASTA+metadata dataset from Gene G haplotypes
    (`msa/phiX174_G.haplotypes.tsv`), partitioning by `mean_pident`/`coverage_pct` into
    reference/divergent-microvirus/etc. "layers."
  - **Phase 1 (`hyphaeon autoclock`)**: CLI call with flags `-a <clean_contigs.fasta>
    -d <clean_metadata.csv> --date-col collection_date --strain-col sequence_id
    --manifold tn93 -H --max-depth 3 --min-leaf-size 20 --min-delta-aicc 15.0
    --n-landmarks auto --max-memory-mb 2048 --output-dir <dir> -o <summary.json>
    -c <classified_metadata.csv>` — a hierarchical, TN93-distance-manifold, spectral/AICc-based
    clustering of taxa into "communities" and "outliers" (no BEAST, no MCMC — closed-form). Output
    JSON schema includes `tmrca`, `rate`, `r2`, `optimal_k`, `delta_aicc`, per-node
    `classification` (e.g. "Chronic / Endemic Reservoir", "Contemporaneous Transmission Dyad").
    Downstream: outlier sequences exported as `quarantined_layer1_outliers.fasta`; each leaf
    community exported as `community_{id}.fasta`.
  - **Phase 2 (Layer 1 quasispecies)**: pure pandas/numpy — computes a "structural potential
    deviation" `delta_phi = -1.2 * |residual|` from the autoclock output, classifies
    "Transient_Deleterious_Unpurged" vs "Permissive_Drift"; re-emits the Gene-E `am3` audit JSON;
    computes per-gene singleton-haplotype fractions from the haplotype TSVs.
  - **Phase 3 (`hyphaeon r0`)**: CLI call `-a <community.fasta> -g <classified_metadata.csv>
    --date-col date --strain-col id --generation-time 0.0174 --generation-sd 0.005 --units days
    -o <r0_summary.json> -c <epoch_skyline_rt.csv>` — closed-form (profile-likelihood, "<5 ms",
    explicitly not BEAST) transmission/phylodynamic R0 and growth-rate estimation. Also
    cross-correlates the Idaho-2024 real-time selection rates (Stage E output) against DMS
    fitness (Spearman/Pearson) and re-emits `idaho2024_realtime_purging_kinetics.csv` and
    `dickins2009_chemostat_trajectories.csv`.
  - **Phase 4 (`hyphaeon meme`)**: CLI call `-a <layer3_speciation_alignment.fasta> --use-tn93
    --attribute --attribution-min-lrt 3.84 -o <meme_attributed.json> -c <meme_sites.csv>` — a
    HyPhy-MEME-like episodic diversifying-selection scan reimplemented in closed form over a
    continuous TN93 distance geometry (explicitly avoids classical codon ML models / HyPhy /
    PAML), followed by a from-scratch **Aggregated Cauchy Association Test (ACAT)** omnibus
    p-value combination (`T = (1/L) Σ tan[(0.5-p_i)π]`) over per-site p-values.
  - **Phase 5+ (`hyphaeon filter`)**: a further CLI subcommand (seen invoked but not read in
    detail) used later in the same script.
  - All phase outputs land under `results/0{0-8}_*/` (see repo layout in Section 4), which are
    exactly the files cited as `Data Dependencies` throughout `sections/*.md`.
- **Important for tool-selection**: because `hyphaeon` subcommands (`autoclock`, `r0`, `meme`,
  `filter`) are a real external CLI with a nontrivial flag surface, any Galaxy wrapper for this
  stage should wrap the **actual `hyphaeon` CLI**, not re-implement its algorithms — but note this
  tool's availability/license/Tool-Shed status is unverified from this repo alone (see Gaps).
- **Reproducibility caveat (important, confirmed by direct script inspection):** not every phase
  in `run_chronaeon_phix174_sieve.py` is a genuine re-executable computation. Phase 0's synthetic
  multi-layer Gene-G FASTA is built with **fabricated, hash-derived pseudo-`collection_date`
  values**, not real SRA/BioSample metadata — any Galaxy reimplementation must source real dates
  or explicitly mark this as a synthetic-metadata demo step, not a data-driven one. Several other
  intermediate statistics that read as "computed" (e.g., the Gene D/E and Gene B/A Mann-Whitney
  medians and p-values cited in Stage H, and a per-gene VEP correlation table) appear as
  **hardcoded constants written directly into the script**, not values recomputed from input data
  at run time. Only Phases 1, 3, and 4 (the `autoclock`, `r0`, and `meme` CLI calls) are confirmed
  genuine, re-executable tool invocations; Phases 2, 6, 7, 8 are largely reporting/munging of
  numbers computed elsewhere (or baked in). A Galaxy translation should treat this whole script as
  an orchestration sketch to be decomposed into real steps, not ported verbatim.

### Stage J — Fane-mechanics and Bull/Wichman literature cross-referencing (pure pandas, no external tool)
- `synthesize_fane_mechanics.py`: joins a curated JSON bibliography (`fane_phage_abstracts.json`,
  61 papers; `fane_ba_papers.json`, 69 papers) against specific DMS table slices by gene/position
  (e.g., Gene D position 61 for the "Gly61 conformational hinge", Gene A domains 1–172 vs
  173–393 vs 394–512 for "A/A* packaging fidelity domain", using `mannwhitneyu` for domain
  contrasts).
- `generate_overlap_trajectory_analysis.py`: the most integrative custom analysis script — combines
  DMS overlap categories (D/E, B/A), Dickins 2009 trajectory positions
  (`G319T,A323G,C324T,A345G,G562T,G570T,G624T,G645T,C656A,...`), and Fane genetics into one
  synthesis, producing `de_overlap_dms_summary.csv` / `ba_overlap_dms_summary.csv` and (per its
  imports) matplotlib/seaborn figures.
- These stages are pure statistics/reporting over already-computed DMS+alignment tables; no
  bioinformatics CLI is shelled out to here.

### Stage K — Figure generation and manuscript QA (project tooling, not science tools)
- `scripts/generate_paper_figures.py`, `generate_comprehensive_figures.py`: matplotlib/seaborn,
  Okabe-Ito colorblind-safe palettes, publication PDF+PNG figure export
  (`paper/figures/fig1_sra_landscape.pdf`, `fig2_sra_spikein_curation.pdf`, etc.), using
  `scipy.stats.{spearmanr,pearsonr,mannwhitneyu}` for in-figure annotations.
- `scripts/generate_dashboard_data.py`: compiles all `results/` + `ref/phix174_proteins.faa` into
  JSON for an offline HTML dashboard (`dashboard/index.html`), not part of the scientific
  pipeline proper.
- `scripts/lint_manuscript_style.py`, `scripts/validate_references.py`,
  `manuscript-authoring-protocol/` (rules/scripts/templates + `PROTOCOL_SUMMARY.md`): a
  **deterministic prose-style linter and citation/DOI validator** enforcing an "anti-LLM
  cliché", specific-cadence writing style (see `scientific_writing.md`) and cross-checking
  citations against CrossRef/PubMed. This is purely an authoring-QA tool, **not relevant to the
  computational/scientific methods** and should be excluded from Galaxy-workflow scope.

## 3. Sample / reference data catalog (for test-data resolution phases)

| Dataset | Identifier(s) | Role | Location in repo |
|---|---|---|---|
| ΦX174 reference genome | RefSeq `NC_001422.1` (5,386 nt) | Universal alignment reference | `ref/NC_001422.1.fasta[.fai]` |
| ΦX174 proteome | 10 genes, `ref/phix174_proteins.faa` | Query panel for LexicMap/kmindex | `ref/`, `cds/*.fasta` |
| Logan SRA petabase index | kmindex: 109 DB shards; LexicMap: 25 domain indices | Universe being mined | external (usegalaxy.org-hosted Galaxy tools) |
| Idaho 2024 chemostat time series | BioProject `PRJNA1174868`, `SRP539859`; runs `SRR31059334`–`SRR31059340` | Real-time selection-rate validation | `data/sra_evolution/idaho2024/` (post-download) |
| Dickins & Nekrutenko 2009 | 10 samples A1-4/B1,3,4/C1,3,4; Galaxy library folder `F175d7d5ddeeb1d43` on usegalaxy.org | Classical chemostat sweep validation | `data/dickins2009/` (post-download); full text `dickins2009_pmc2817424.xml` |
| Wei, Li & Lehner 2026 DMS | bioRxiv 10.64898/2026.07.25.740675; N=39,335 variants / 16,098 SNVs | Ground-truth fitness surface for everything | **NOT present in repo** — expected at `phix174_WGM/data/...zip` |
| NCBI Virus complete genomes | 185 complete GenBank records | "91.8% derive from lab evolution" certification | `data/ncbi_virus_*` |
| Nominal-taxonomy SRA audit | TaxID `10847` / `2886930`; 18 BioProjects incl. `SRP042938`,`SRP008975`,`ERP002532`,`ERP000074`,`SRP001260`,`ERP001254`,`SRP347090`,`SRP653402`,`ERP001281`,`SRP066453`,`SRP059208`,`SRP082602`,`SRP009433`,`ERP000177`,`SRP003315`,`SRP363953`,`DRP008496` | Provenance certification / negative-control catalog | `paper/results_section1_sra_landscape.md`, Table 2 in the .tex |
| Fane genetics bibliography | 61–69 curated papers | Mechanistic cross-referencing | `fane_phage_abstracts.json`, `fane_ba_papers.json` |
| EVcouplings MSAs | author-supplied, from Figshare | HyphAeon input alignments | `msas/`, `msas_sub/` (350-taxa subsampled) |
| Structural references | PDB 1CD3 / 2BPA (cryo-EM coordinates) | Structural context for Fane mechanics, cited not computed on | none locally (citation only) |

## 4. `results/` directory shape (already-computed intermediate/output tables — useful as
   expected-output fixtures for a Galaxy test plan)

```
results/00_ingestion/            lexicmap_9genes_ingestion_manifest.csv, qc_clean_contigs.fasta, qc_clean_metadata.csv
results/01_autoclock_deconvolution/  autoclock_summary.json, hierarchical_summary.json, hierarchical_tree.json,
                                      classified_taxa_metadata.csv, contemporaneous_dyads.csv
results/02_layer1_quasispecies/   gene_e_am3_quarantine_audit.json, quasispecies_cloud.csv, purifying_selection_gradient.csv
results/03_layer2_transmission/   dickins2009_chemostat_trajectories.csv, idaho2024_realtime_purging_kinetics.csv, r0_phylodynamics_summary.json
results/04_layer3_speciation/     bull_wichman_adaptive_sites_benchmark.csv, hyphaeon_meme_attributed.json,
                                   hyphaeon_meme_sites.csv, layer3_speciation_alignment.fasta, neural_lineage_selection_drivers.csv
results/05_layer4_superfamily/    deep_structural_invariants.csv, structural_skeleton.fasta
results/06_dms_benchmarking/      proteome_vep_benchmark.csv
results/07_overlapping_frames/    ba_overlap_dms_summary.csv, de_overlap_dms_summary.csv, shadow_effect_calibration.json
results/08_fane_mechanics/        fane_mechanistic_landmarks.csv, gene_a_domain_dms_summary.csv, interface_kde_curves.csv
```
This numbered-layer structure (00 ingestion → 01 autoclock → 02 layer1 → 03 layer2 → 04 layer3 →
05 layer4 → 06 benchmarking → 07 overlaps → 08 Fane) is a strong hint for how to decompose the
eventual Galaxy workflow into stages/subworkflows.

## 5. Tool inventory — real/installable vs. custom/internal

**Real, externally installable tools/CLIs actually invoked in code:**
- `kmindex` (Galaxy Tool Shed: `iuc/kmindex/kmindex_query/0.6.1+galaxy4`) — k-mer containment search
- `LexicMap` (Galaxy Tool Shed: `iuc/lexicmap/lexicmap_search/0.9.0+galaxy1`) — sequence-to-graph streaming aligner
- `bowtie2` — short-read alignment (Idaho 2024, Dickins 2009)
- `samtools` (`sort`, `index`, `mpileup`) — BAM handling and pileup generation
- `FastTree` — maximum-likelihood nucleotide phylogeny (GTR model) on codon MSAs
- `logan-walker` (Rust binary, `nekrut/disassembler` project) — cDBG graph traversal / intra-host variant calling; conceptually plays the role of a reference-free variant caller / graph aligner
- `hyphaeon` — real CLI (`autoclock`, `r0`, `meme`, `filter` subcommands) + real Python package (`hyphaeon.inference`, `hyphaeon.disease`) — phylogenetic transformer VEP + closed-form phylodynamics/selection-scan toolkit (`github.com/veg/hyphaeon`)
- `hyphaeon_overlaps` — companion Python package for overlapping-ORF consequence classification (`nekrut/hyphaeon-overlaps[-cdx]`)
- `BioBlend` / Galaxy REST API — orchestration glue for all usegalaxy.org-hosted steps
- Standard Python scientific stack: `pandas`, `numpy`, `scipy.stats` (spearmanr, pearsonr, mannwhitneyu, ks_2samp, ranksums), `Biopython` (`SeqIO`, `Seq`, `CodonTable`), `torch`, `matplotlib`/`seaborn`
- Comparison-only VEP tools (scores consumed from pre-merged tables, not re-run here): GEMME, ProteoCast, ESCOTT, RSALOR, DDMut, ESM-2 (15B), ESM3 Open, Tranception-L, ESM_IF1, MIF_ST

**Custom/internal orchestration code (no separate installable tool exists — would need to be
Galaxy-wrapped as bespoke scripts, or reimplemented as native Galaxy steps):**
- `stream_lexicmap_msa.py` — multi-HSP coordinate tiling, codon QC, haplotype collapsing (the
  paper's key methodological contribution; likely the single most important custom "tool" to
  wrap for Galaxy, since it has no off-the-shelf equivalent)
- `run_chronaeon_phix174_sieve.py` — orchestration only ("ChronAeon" = a name for this script's
  phase sequence + the ACAT combination test, not a separate binary)
- Custom pileup parser embedded in `analyze_idaho_evolution.py` / `call_dickins_trajectories.py`
  (hand-rolled samtools-mpileup string parser — could likely be replaced by a standard VCF
  caller such as `bcftools mpileup`/`lofreq`/`varscan` in a Galaxy-native reimplementation)
- `create_targeted_diversity_cohorts.py`, `synthesize_fane_mechanics.py`,
  `generate_overlap_trajectory_analysis.py`, `calibrate_hyphaeon_shadow_effect.py` — bespoke
  pandas/scipy statistics scripts with no external tool dependency beyond the DMS table

**Names that are NOT separate tools** (flagging per task instructions): "ChronAeon" is a
project/spec-document name for an orchestration pattern (a fixed sequence of `hyphaeon` CLI
calls + a custom ACAT step), not an importable package or binary distinct from `hyphaeon`
itself. "Fane mechanics" / "Fane morphogenesis" refers to the body of work by virologist
Bentley A. Fane (cited literature), not a tool. "AM3 sieve" / "am3 quarantine" refers to the
biological `am3` mutant + the QC filtering logic inside `stream_lexicmap_msa.py`, not a separate
program.

## 6. Key literature / reference data anchors (for citation/README generation downstream)
- Sanger et al. 1977, *Nature* 265:687-695 (doi:10.1038/265687a0) — first DNA genome, `am3` strain origin
- Bull et al. 1997, *Genetics* 147:1497-1507 — thermal adaptation, parallel convergence
- Wichman et al. 1999, *Science* 285:422-424 — parallel evolutionary trajectories
- Dickins & Nekrutenko 2009, *Genome Biol. Evol.* 1:294-307 (doi:10.1093/gbe/evp029) — chemostat trajectories
- Wei, Li & Lehner 2026, bioRxiv doi:10.64898/2026.07.25.740675 — whole-genome/proteome DMS ground truth
- Fane & Hayashi 1991; Chen/Uchiyama/Fane 2007; Cherwa/Uchiyama/Fane 2008 (PMID 18400861); Cherwa et al. 2011; Hafenstein & Fane 2002; Roznowski et al. 2020 (PMID 31666371) — morphogenetic/structural genetics
- Lin et al. 2023, *Science* 379:1123-1130 — ESM-2 protein language model
- Mitra et al. 2015; Kozich et al. 2013 — Illumina PhiX spike-in practice, cited for the 0.5–5% spike-in-concentration claim
- Full BibTeX: `paper/references_canonical.bib`

## 7. Assumptions and open gaps (carried forward per skill protocol)

1. **draft_manuscript.tex is materially narrower than the full project.** Treated the .tex file
   as authoritative for "what the paper claims" and everything else as context/rationale, per
   explicit task instruction. A later phase should confirm whether the Galaxy workflow target is
   scoped to *only* the SRA-landscape/spike-in pipeline (Stages A–D, D2) actually in the current
   .tex, or the full multi-stage project (through HyphAeon/Fane synthesis) implied by
   HANDOVER.md/sections. This summary documents both but does not resolve the scoping question.
2. **The Wei et al. 2026 DMS dataset is a hard external dependency not present in this repo**
   (`phix174_WGM/...zip`, expected at a path that doesn't exist locally). Any reproducible
   workflow needs this sourced from the paper's data-availability statement / bioRxiv supplement.
3. **HyphAeon's installability/licensing/Tool-Shed status is unverified.** Confirmed it's a real
   CLI+library (`github.com/veg/hyphaeon` per citations) invoked via subprocess/import, but this
   summary could not check whether it's conda/pip-installable or Tool-Shed-wrapped already —
   important for the Galaxy tool-discovery phase.
4. **The host/biome/platform metadata stratification of the 2.1M-accession union set** (Figure 1B/C
   numbers in the paper) has no corresponding script among the root-level `.py` files found;
   likely produced via ad hoc NCBI Entrez/SRA-metadata queries or manual curation not captured in
   this repo snapshot. Flagged as a reproducibility gap rather than invented.
5. Several absolute paths are hardcoded to the original author's machine (`/home/anton/...`,
   `/Users/anton/...`, `/Users/sergei/...`, `/tmp/gxy.txt` API key) — treat all such paths as
   illustrative/non-portable; a Galaxy workflow would need these parameterized as inputs/config.
6. `manuscript-authoring-protocol/` was checked and confirmed to be purely about prose style/
   citation-hygiene tooling (per instructions), not science methods — excluded from the pipeline
   description above.
7. `run_chronaeon_phix174_sieve.py` (Stage I) mixes genuine tool calls (Phases 1/3/4, real
   `hyphaeon` CLI invocations) with a synthetic-metadata demo step (Phase 0's fabricated
   pseudo-dates) and several hardcoded/pre-baked statistics in the later phases. Treat this script
   as evidence of *intended* pipeline shape, not as a script to port verbatim into Galaxy steps —
   flagged so a later phase does not mistake baked-in constants for a computation to reproduce.
8. Only two stages are confirmed to already run as installed Galaxy Tool Shed tools today
   (`kmindex_query` 0.6.1+galaxy4, `lexicmap_search` 0.9.0+galaxy1, both via BioBlend against
   usegalaxy.org). Everything else in the pipeline (multi-HSP tiling/QC, disassembler/logan-walker,
   mpileup trajectory calling, HyphAeon, hyphaeon_overlaps, FastTree, ChronAeon orchestration) is
   currently ad hoc local/BioBlend scripting and will need new Galaxy tool wrappers or reimplementation
   as native Galaxy steps.

## 8. Foundry feedback

No feedback ledger entry was appended for this run. Rationale: nothing encountered during this
extraction pointed at a defect, gap, or self-contradiction in the `summarize-paper` skill bundle
itself (its SKILL.md and `_feedback.md` protocol were both clear and sufficient to complete this
task). The difficulties encountered — a narrower-than-expected primary paper, an external dataset
missing from the repo, an unscripted metadata-stratification step — are properties of this
specific project's source material, not of the Foundry skill asset, so per the ledger's own
scope rule ("does not record unmet requirements in the workflow being built") they do not
qualify as ledger entries; they are instead recorded above in Section 7 as assumptions/gaps for
the next pipeline phase to consume.

plan_version: "1"

source:
  kind: freeform
  name: "ΦX174 Planetary-Scale Retrospective DMS -- Workflow A (SRA Landscape & Spike-In Sieve)"
  derived_from: intent
  notes: >-
    Synthesized from freeform-summary.md (Stages A, B, C, D2), freeform-galaxy-interface.md
    (Workflow A section), freeform-galaxy-data-flow.md (nodes N1-N7), and
    iwc-comparison-notes.md (no High/Medium-confidence IWC domain exemplar found; kmindex and
    LexicMap do not appear anywhere in the IWC corpus). There is no upstream test-evidence
    (nf-test snapshot, CWL job file) for this project to translate -- every test case and
    assertion below is synthesized from stated workflow intent, the paper's own Table 1 /
    am3-diagnostic numbers, and the concrete workflow's declared step contracts. Grounded
    additionally against galaxy-workflow.gxwf.yml (the concrete 9-step gxformat2 draft) and
    test-data-refs.json (phase 7's gene E/J scoping decision and documented fixture gaps), per
    this run's explicit instruction to use them for consistency even though they are not this
    skill's own declared inputs.

workflow:
  title: "phix174_sra_landscape_spikein_sieve (ΦX174 SRA Landscape & Spike-In Sieve, Workflow A)"
  label_source: draft
  notes: >-
    All workflow_label values below are the literal input/output ids from
    galaxy-workflow.gxwf.yml's inputs:/outputs: blocks (e.g. gene_query_panel,
    gene_e_am3_quarantine_audit), not the interface brief's earlier proposed prose labels
    (e.g. "Gene E am3 quarantine audit"), which in places differ from what the concrete draft
    actually shipped. implement-galaxy-workflow-test should still re-confirm these against the
    live draft before finalizing, since this skill's own declared inputs are the earlier
    template-era briefs (see warnings[0]).

test_cases:
  - id: kmindex_wiring_smoke_generic_fixtures
    doc: >-
      Structural/wiring smoke test for the kmindex containment-screen chain
      (combine_gene_panel_to_bulk_fasta -> kmindex_containment_screen -> kmindex_hit_concat ->
      kmindex_hit_dedup_max_score) and for lexicmap_search's multi-index-selection binding,
      using each pinned Tool Shed wrapper's own real upstream functional-test fixtures (generic,
      non-phiX174 sequences and indices) instead of phiX174 biology. This exists because no
      small, real, locally-buildable Logan-shard kmindex index or Logan-derived LexicMap domain
      index exists at any scale smaller than usegalaxy.org's production data (test-data-refs.json
      gaps "no-real-small-kmindex-logan-shard-subset" and
      "no-real-small-lexicmap-logan-index-subset"). This case proves the workflow's collection
      map-over, multi-select DB/index binding, and per-gene sensitivity-override wiring execute
      without error; it does NOT audit any biological content, including the Gene E am3
      diagnostic (see test case gene_e_j_am3_diagnostic_synthetic_lexicmap_index for that).
      The workflow's mandatory element_identifier=="E" extraction in extract_gene_e_am3_audit
      means every test case, including this one, must still supply an "E"-identified
      gene_query_panel element, even though here it carries generic (non-phiX174) sequence
      content.
    derived_from: intent
    provenance: >-
      test-data-refs.json tool_level_structural_fixtures (kmindex_query test #6 "using register
      index"; lexicmap.xml tests #3/#4/#6); iwc-comparison-notes.md "Test issues" routing note.
    job_inputs:
      - workflow_label: gene_query_panel
        label_status: resolved
        description: >-
          2-element sample_sheet reusing the workflow's mandatory E/J identifiers but populated
          with each pinned tool's own generic upstream test query sequences, not real phiX174
          CDS -- element "E" holds tools-iuc kmindex's query1.fasta content (or lexicmap's
          lexicmap_query3.fasta content), element "J" a second generic sequence from the same
          fixture sets. Per-gene LexicMap sensitivity override columns left unset for both
          (not meaningful for generic sequences).
        collection_shape: sample_sheet
        datatype: fasta
        fixture:
          storage: remote-url
          location: >-
            https://raw.githubusercontent.com/galaxyproject/tools-iuc/main/tools/kmindex/test-data/query1.fasta
            ; https://raw.githubusercontent.com/galaxyproject/tools-iuc/main/tools/lexicmap/test-data/lexicmap_query3.fasta
          checksum: null
          provenance: >-
            tools-iuc kmindex_query.xml test #6 and lexicmap.xml tests #3/#4/#6, both already
            used for upstream CI; not phiX174 biology.
      - workflow_label: kmindex_db_selection
        label_status: resolved
        description: >-
          kmindex's own test-only "register" value, which expands (in the wrapper's own
          functional test) to a real 2-shard multi-select (index1, index2) against a synthetic
          repeat-sequence index bundled in tools-iuc's test-data.
        collection_shape: null
        datatype: null
        fixture:
          storage: unresolved
          location: null
          checksum: null
          provenance: >-
            Value "register" is confirmed only inside kmindex_query.xml's own functional test
            harness; whether a workflow-level Planemo test job can pass it through the same way
            is unconfirmed -- see unresolved[2].
      - workflow_label: kmindex_zvalue
        label_status: resolved
        description: workflow default.
        collection_shape: null
        datatype: null
        fixture: {storage: null, location: "6", checksum: null, provenance: "galaxy-workflow.gxwf.yml default"}
      - workflow_label: kmindex_threshold
        label_status: resolved
        description: workflow default.
        collection_shape: null
        datatype: null
        fixture: {storage: null, location: "0.3", checksum: null, provenance: "galaxy-workflow.gxwf.yml default"}
      - workflow_label: kmindex_output_format
        label_status: resolved
        description: workflow default.
        collection_shape: null
        datatype: null
        fixture: {storage: null, location: "json", checksum: null, provenance: "galaxy-workflow.gxwf.yml default"}
      - workflow_label: kmindex_fast
        label_status: resolved
        description: workflow default.
        collection_shape: null
        datatype: null
        fixture: {storage: null, location: "false", checksum: null, provenance: "galaxy-workflow.gxwf.yml default"}
      - workflow_label: lexicmap_index_selection
        label_status: resolved
        description: >-
          LexicMap's own real 2-index test combination (db.lmi + db2.lmi), indexing two small
          real viral RefSeq assemblies (GCF_001502155.1, GCF_001502175.1) -- not phiX174 and not
          Logan-derived.
        collection_shape: null
        datatype: null
        fixture:
          storage: unresolved
          location: null
          checksum: null
          provenance: >-
            tools-iuc lexicmap.xml tests #3/#4/#6 combine db.lmi+db2.lmi; whether the exact
            data-table name is reachable from a workflow-level test job is unconfirmed -- see
            unresolved[2].
      - workflow_label: lexicmap_top_n_genomes
        label_status: resolved
        description: workflow default.
        collection_shape: null
        datatype: null
        fixture: {storage: null, location: "0", checksum: null, provenance: "galaxy-workflow.gxwf.yml default"}
      - workflow_label: lexicmap_advanced_all
        label_status: resolved
        description: workflow default.
        collection_shape: null
        datatype: null
        fixture: {storage: null, location: "true", checksum: null, provenance: "galaxy-workflow.gxwf.yml default"}
      - workflow_label: tiling_qc_min_coverage
        label_status: resolved
        description: workflow default.
        collection_shape: null
        datatype: null
        fixture: {storage: null, location: "0.8", checksum: null, provenance: "galaxy-workflow.gxwf.yml default"}
      - workflow_label: tiling_qc_min_coverage_partial
        label_status: resolved
        description: workflow default.
        collection_shape: null
        datatype: null
        fixture: {storage: null, location: "0.5", checksum: null, provenance: "galaxy-workflow.gxwf.yml default"}
      - workflow_label: tiling_qc_min_pident
        label_status: resolved
        description: workflow default.
        collection_shape: null
        datatype: null
        fixture: {storage: null, location: "60", checksum: null, provenance: "galaxy-workflow.gxwf.yml default"}
      - workflow_label: tiling_qc_max_internal_stops
        label_status: resolved
        description: workflow default.
        collection_shape: null
        datatype: null
        fixture: {storage: null, location: "0", checksum: null, provenance: "galaxy-workflow.gxwf.yml default"}
      - workflow_label: tiling_qc_sample_cap
        label_status: resolved
        description: workflow default.
        collection_shape: null
        datatype: null
        fixture: {storage: null, location: "10", checksum: null, provenance: "galaxy-workflow.gxwf.yml default"}
      - workflow_label: tiling_qc_allow_frameshifts
        label_status: resolved
        description: >-
          Confirmed OFF/False by direct read of nekrut/disassembler's lexicmap_streamer.py
          argparse (store_true, no default=True).
        collection_shape: null
        datatype: null
        fixture: {storage: null, location: "false", checksum: null, provenance: "test-data-refs.json inputs (tiling_qc_allow_frameshifts)"}
    expected_outputs:
      - workflow_label: kmindex_accession_union
        label_status: resolved
        description: >-
          Deduplicated accession union across the kmindex "register" test shards. Existence-only:
          the underlying index has no phiX174 biological relationship, so only "the chain ran and
          produced non-empty output" is assertable.
        output_kind: dataset
        collection_shape: null
        assertion_intent:
          - family: has_size
            intent: "Output is non-empty (chain executed and produced a result)."
            expected_value: null
            tolerance: {kind: none, magnitude: null, rationale: "existence-only, no content relationship to phiX174"}
            element_identifier: null
            evidence: intent
            confidence: low
      - workflow_label: clean_full_length_cds_haplotypes
        label_status: resolved
        description: "Per-gene clean haplotype FASTA (sample_sheet). Existence-only per element."
        output_kind: collection
        collection_shape: sample_sheet
        assertion_intent:
          - {family: has_size, intent: "Non-empty output for gene E.", expected_value: null, tolerance: null, element_identifier: "E", evidence: intent, confidence: low}
          - {family: has_size, intent: "Non-empty output for gene J.", expected_value: null, tolerance: null, element_identifier: "J", evidence: intent, confidence: low}
      - workflow_label: clean_haplotype_counts
        label_status: resolved
        description: "Per-gene clean haplotype counts table. Existence-only per element."
        output_kind: collection
        collection_shape: sample_sheet
        assertion_intent:
          - {family: has_n_columns, intent: "Table has a plausible column count (schema smoke check).", expected_value: null, tolerance: null, element_identifier: "E", evidence: intent, confidence: low}
          - {family: has_n_columns, intent: "Table has a plausible column count (schema smoke check).", expected_value: null, tolerance: null, element_identifier: "J", evidence: intent, confidence: low}
      - workflow_label: flagged_accessions
        label_status: resolved
        description: "Per-gene flagged (QC-failed) sequences. Existence-only, not audited for content in this case."
        output_kind: collection
        collection_shape: sample_sheet
        assertion_intent: []
      - workflow_label: flagged_audit_reasons
        label_status: resolved
        description: "Per-gene flagged.tsv. Existence-only in this case; see the other test case for the real am3 assertion."
        output_kind: collection
        collection_shape: sample_sheet
        assertion_intent:
          - {family: has_n_lines, intent: "At least a header line is present.", expected_value: null, tolerance: {kind: none, magnitude: null, rationale: "existence-only"}, element_identifier: "E", evidence: intent, confidence: low}
      - workflow_label: per_gene_ingestion_summary
        label_status: resolved
        description: "Per-gene summary.json. Stochastic/opaque given generic non-phiX174 input; existence-only per corpus convention for JSON of this kind."
        output_kind: collection
        collection_shape: sample_sheet
        assertion_intent:
          - {family: has_text, intent: "Output is well-formed JSON.", expected_value: "{", tolerance: null, element_identifier: "E", evidence: intent, confidence: low}
          - {family: has_text, intent: "Output is well-formed JSON.", expected_value: "{", tolerance: null, element_identifier: "J", evidence: intent, confidence: low}
      - workflow_label: ingestion_manifest_all_genes
        label_status: resolved
        description: "All-gene manifest table. Existence-only (2 data rows expected structurally, not content-checked)."
        output_kind: dataset
        collection_shape: null
        assertion_intent:
          - {family: has_n_lines, intent: "Header plus 2 gene rows (E, J).", expected_value: 3, tolerance: {kind: delta, magnitude: 1, rationale: "tolerate a trailing-newline off-by-one"}, element_identifier: null, evidence: intent, confidence: medium}
      - workflow_label: gene_e_am3_quarantine_audit
        label_status: resolved
        description: >-
          Gene E element of flagged_tsv. In THIS case the "E" element is generic test-fixture
          sequence, not real phiX174 gene E, so no am3 diagnostic content is expected --
          existence-only. The meaningful, content-bearing am3 assertion is in the other test
          case.
        output_kind: dataset
        collection_shape: null
        assertion_intent:
          - family: has_size
            intent: "Output exists (extraction step resolved a real E element and did not error)."
            expected_value: null
            tolerance: {kind: none, magnitude: null, rationale: "existence-only; not phiX174 biology in this case"}
            element_identifier: null
            evidence: intent
            confidence: low

  - id: gene_e_j_am3_diagnostic_synthetic_lexicmap_index
    doc: >-
      Primary functional test for Workflow A's core scientific claim (the paper's Sanger am3
      spike-in diagnostic, freeform-summary.md Stage D2: genome position nt587 G->A causes a
      premature TGG->TAG stop at Gene E codon 7, gpE_W7*). Exercises lexicmap_search ->
      lexicmap_streamer_tiling_qc -> {flatten_gene_summary_json_to_row ->
      join_gene_summary_rows_into_manifest} and -> extract_gene_e_am3_audit end-to-end using a
      small, purpose-built synthetic LexicMap domain index (construction deferred to
      implement-galaxy-workflow-test / a follow-on find-test-data pass; this plan specifies its
      required composition per Fixture.storage=generated-toy) seeded with real cds/E.fasta and
      cds/J.fasta content plus 3 hand-crafted decoy accessions: (1) a Gene-E wildtype-clean
      decoy, (2) a Gene-E am3-positive decoy carrying the exact nt587 G->A / codon-7 TGG->TAG
      substitution, and (3) a Gene-J clean decoy. This is the smallest fixture that can produce a
      real, assertable Gene E am3 quarantine audit without waiting on usegalaxy.org-hosted
      production Logan indices (test-data-refs.json gap
      "no-real-completed-lexicmap-hits-table-for-lexicmap-streamer"). The kmindex chain shares
      the same job (one workflow invocation runs both chains) but is not the focus here and is
      only weakly asserted; see omissions[1].
    derived_from: intent
    provenance: >-
      freeform-summary.md Stage D2 (am3 diagnostic: nt587 G->A, gpE_W7*, 2,215,172/2,392,457 =
      92.59% of evaluated Gene E accessions); test-data-refs.json inputs[0] (real cds/E.fasta,
      cds/J.fasta, per-gene sensitivity overrides) and gaps[2]; galaxy-workflow.gxwf.yml
      extract_gene_e_am3_audit doc (flagged_tsv columns: accession, coverage, mean_pident, flag,
      n_stops, stop_codons, hsps_merged).
    job_inputs:
      - workflow_label: gene_query_panel
        label_status: resolved
        description: >-
          2-element sample_sheet: E = real cds/E.fasta (273 bp CDS) with per-gene LexicMap
          sensitivity override columns populated (align_min_match_pident=60.0,
          align_min_match_len=35, seed_min_prefix=15, min_qcov_per_genome=30.0, sourced from
          logan_remaining_runs.json per test-data-refs.json); J = real cds/J.fasta (114 bp CDS,
          shortest gene in the panel), override columns unset (falls back to wrapper defaults
          70/50/17/unset).
        collection_shape: sample_sheet
        datatype: fasta
        fixture:
          storage: in-repo
          location: "cds/E.fasta, cds/J.fasta"
          checksum: null
          provenance: >-
            test-data-refs.json inputs[0]; real files already used by this project's own
            kmindex/LexicMap submission scripts (submit_kmindex_all_genes.py,
            submit_remaining_logan.py).
      - workflow_label: kmindex_db_selection
        label_status: resolved
        description: "Real Logan shard names, kept only for job completeness; not this case's focus."
        collection_shape: null
        datatype: null
        fixture:
          storage: unresolved
          location: "GENOMIC_PHG,GENOMIC_VRL,METAGENOMIC_ENV"
          checksum: null
          provenance: >-
            test-data-refs.json kmindex_db_selection (real shard names, no backing test-scale
            index -- see unresolved[0]).
      - workflow_label: kmindex_zvalue
        label_status: resolved
        description: workflow default.
        collection_shape: null
        datatype: null
        fixture: {storage: null, location: "6", checksum: null, provenance: "galaxy-workflow.gxwf.yml default"}
      - workflow_label: kmindex_threshold
        label_status: resolved
        description: workflow default.
        collection_shape: null
        datatype: null
        fixture: {storage: null, location: "0.3", checksum: null, provenance: "galaxy-workflow.gxwf.yml default"}
      - workflow_label: kmindex_output_format
        label_status: resolved
        description: workflow default.
        collection_shape: null
        datatype: null
        fixture: {storage: null, location: "json", checksum: null, provenance: "galaxy-workflow.gxwf.yml default"}
      - workflow_label: kmindex_fast
        label_status: resolved
        description: workflow default.
        collection_shape: null
        datatype: null
        fixture: {storage: null, location: "false", checksum: null, provenance: "galaxy-workflow.gxwf.yml default"}
      - workflow_label: lexicmap_index_selection
        label_status: assumed
        description: >-
          Placeholder data-table name for the not-yet-built synthetic 3-decoy index described in
          this test case's doc (e.g. "PhiX174E_J_Am3ToyIndex"); real value to be assigned once
          the index is constructed.
        collection_shape: null
        datatype: null
        fixture:
          storage: generated-toy
          location: null
          checksum: null
          provenance: >-
            This plan specifies the index's required composition (wildtype-clean Gene E decoy,
            am3-positive Gene E decoy, clean Gene J decoy); construction is deferred -- see
            unresolved[1].
      - workflow_label: lexicmap_top_n_genomes
        label_status: resolved
        description: workflow default.
        collection_shape: null
        datatype: null
        fixture: {storage: null, location: "0", checksum: null, provenance: "galaxy-workflow.gxwf.yml default"}
      - workflow_label: lexicmap_advanced_all
        label_status: resolved
        description: workflow default.
        collection_shape: null
        datatype: null
        fixture: {storage: null, location: "true", checksum: null, provenance: "galaxy-workflow.gxwf.yml default"}
      - workflow_label: tiling_qc_min_coverage
        label_status: resolved
        description: workflow default.
        collection_shape: null
        datatype: null
        fixture: {storage: null, location: "0.8", checksum: null, provenance: "galaxy-workflow.gxwf.yml default"}
      - workflow_label: tiling_qc_min_coverage_partial
        label_status: resolved
        description: workflow default.
        collection_shape: null
        datatype: null
        fixture: {storage: null, location: "0.5", checksum: null, provenance: "galaxy-workflow.gxwf.yml default"}
      - workflow_label: tiling_qc_min_pident
        label_status: resolved
        description: workflow default.
        collection_shape: null
        datatype: null
        fixture: {storage: null, location: "60", checksum: null, provenance: "galaxy-workflow.gxwf.yml default"}
      - workflow_label: tiling_qc_max_internal_stops
        label_status: resolved
        description: >-
          workflow default; 0 tolerated internal stops for the CLEAN cohort is exactly what
          routes the am3-positive decoy to the FLAGGED cohort instead.
        collection_shape: null
        datatype: null
        fixture: {storage: null, location: "0", checksum: null, provenance: "galaxy-workflow.gxwf.yml default"}
      - workflow_label: tiling_qc_sample_cap
        label_status: resolved
        description: workflow default.
        collection_shape: null
        datatype: null
        fixture: {storage: null, location: "10", checksum: null, provenance: "galaxy-workflow.gxwf.yml default"}
      - workflow_label: tiling_qc_allow_frameshifts
        label_status: resolved
        description: "Confirmed OFF/False from nekrut/disassembler's real argparse source."
        collection_shape: null
        datatype: null
        fixture: {storage: null, location: "false", checksum: null, provenance: "test-data-refs.json inputs (tiling_qc_allow_frameshifts)"}
    expected_outputs:
      - workflow_label: kmindex_accession_union
        label_status: resolved
        description: "Not this case's focus; see omissions[1]."
        output_kind: dataset
        collection_shape: null
        assertion_intent: []
      - workflow_label: clean_full_length_cds_haplotypes
        label_status: resolved
        description: >-
          Per-gene clean haplotype FASTA. Gene E should contain exactly the wildtype-clean decoy
          (the am3-positive decoy is routed to FLAGGED, not here). Gene J should contain its one
          clean decoy.
        output_kind: collection
        collection_shape: sample_sheet
        assertion_intent:
          - family: has_text
            intent: "Gene E's clean cohort contains the wildtype-clean decoy accession, not the am3-positive one."
            expected_value: "E_wildtype_ctrl"
            tolerance: null
            element_identifier: "E"
            evidence: intent
            confidence: medium
          - family: has_text
            intent: "Gene J's clean cohort contains its clean decoy accession."
            expected_value: "J_clean_ctrl"
            tolerance: null
            element_identifier: "J"
            evidence: intent
            confidence: medium
      - workflow_label: clean_haplotype_counts
        label_status: resolved
        description: "Per-gene clean haplotype counts. Gene E: 1 clean haplotype. Gene J: 1 clean haplotype."
        output_kind: collection
        collection_shape: sample_sheet
        assertion_intent:
          - {family: has_n_lines, intent: "Header plus exactly 1 clean haplotype row for gene E.", expected_value: 2, tolerance: {kind: delta, magnitude: 1, rationale: "tolerate trailing-newline off-by-one"}, element_identifier: "E", evidence: intent, confidence: medium}
          - {family: has_n_lines, intent: "Header plus exactly 1 clean haplotype row for gene J.", expected_value: 2, tolerance: {kind: delta, magnitude: 1, rationale: "tolerate trailing-newline off-by-one"}, element_identifier: "J", evidence: intent, confidence: medium}
      - workflow_label: flagged_accessions
        label_status: resolved
        description: "Per-gene flagged (QC-failed) sequences. Gene E should contain exactly the am3-positive decoy. Gene J should be empty."
        output_kind: collection
        collection_shape: sample_sheet
        assertion_intent:
          - family: has_text
            intent: "Gene E's flagged cohort contains the am3-positive decoy accession."
            expected_value: "E_am3_ctrl"
            tolerance: null
            element_identifier: "E"
            evidence: intent
            confidence: medium
      - workflow_label: flagged_audit_reasons
        label_status: resolved
        description: >-
          Per-gene flagged.tsv (columns: accession, coverage, mean_pident, flag, n_stops,
          stop_codons, hsps_merged, per galaxy-workflow.gxwf.yml's confirmed real source read).
          This is the closest upstream signal to the am3 finding before extraction.
        output_kind: collection
        collection_shape: sample_sheet
        assertion_intent:
          - family: has_n_lines
            intent: "Gene E: header plus exactly 1 flagged row (the am3-positive decoy)."
            expected_value: 2
            tolerance: {kind: delta, magnitude: 1, rationale: "tolerate trailing-newline off-by-one"}
            element_identifier: "E"
            evidence: intent
            confidence: medium
          - family: has_text
            intent: "Gene E's flagged row records the premature stop at codon 7, matching the paper's gpE_W7* diagnostic."
            expected_value: "7"
            tolerance: null
            element_identifier: "E"
            evidence: intent
            confidence: medium
          - family: has_n_lines
            intent: "Gene J: header only, no flagged accessions (clean decoy only)."
            expected_value: 1
            tolerance: {kind: delta, magnitude: 0, rationale: "exact -- this fixture is fully controlled/deterministic by design"}
            element_identifier: "J"
            evidence: intent
            confidence: medium
      - workflow_label: per_gene_ingestion_summary
        label_status: resolved
        description: >-
          Per-gene summary.json. Deterministic here (tiny controlled fixture, not the
          stochastic/floating-point-heavy case the existence-only convention targets), so a
          surgical property assertion is used instead of "starts with {".
        output_kind: collection
        collection_shape: sample_sheet
        assertion_intent:
          - family: has_json_property_with_value
            intent: "Gene E evaluated exactly 2 accessions (wildtype-clean + am3-positive decoys)."
            expected_value: 2
            tolerance: null
            element_identifier: "E"
            evidence: intent
            confidence: medium
          - family: has_json_property_with_value
            intent: "Gene E's cohort breakdown records exactly 1 premature-stop (am3-like) flagged accession."
            expected_value: 1
            tolerance: null
            element_identifier: "E"
            evidence: intent
            confidence: medium
          - family: has_json_property_with_value
            intent: "Gene J evaluated exactly 1 accession (its one clean decoy)."
            expected_value: 1
            tolerance: null
            element_identifier: "J"
            evidence: intent
            confidence: medium
      - workflow_label: ingestion_manifest_all_genes
        label_status: resolved
        description: "All-gene manifest table joining the E and J summary rows."
        output_kind: dataset
        collection_shape: null
        assertion_intent:
          - family: has_n_lines
            intent: "Header plus 2 gene rows (E, J)."
            expected_value: 3
            tolerance: {kind: delta, magnitude: 1, rationale: "tolerate trailing-newline off-by-one"}
            element_identifier: null
            evidence: intent
            confidence: medium
          - family: has_text
            intent: "Manifest's first column is the gene symbol (per flatten_gene_summary_json_to_row's own confirmed column-1-is-gene contract)."
            expected_value: "gene"
            tolerance: null
            element_identifier: null
            evidence: intent
            confidence: medium
      - workflow_label: gene_e_am3_quarantine_audit
        label_status: resolved
        description: >-
          FLAGSHIP ASSERTION. Gene E element of flagged_tsv, extracted by
          extract_gene_e_am3_audit (__EXTRACT_DATASET__, by_identifier "E"). With this test
          case's synthetic index, this should contain exactly one row: the am3-positive decoy,
          flagged for a premature stop at codon 7 -- the workflow-level, assertable analog of the
          paper's real finding that 2,215,172/2,392,457 (92.59%) of evaluated Gene E accessions
          carry the nt587 G->A / gpE_W7* substitution. Real full-scale numbers are recorded only
          as directional context in test-data-refs.json and are NOT asserted here (see
          omissions[0]).
        output_kind: dataset
        collection_shape: null
        assertion_intent:
          - family: has_n_lines
            intent: "Exactly one flagged row (header plus the single am3-positive decoy) for the tiny controlled fixture."
            expected_value: 2
            tolerance: {kind: delta, magnitude: 0, rationale: "exact -- this output is a direct extraction of a fully controlled synthetic fixture, not a stochastic tool result"}
            element_identifier: null
            evidence: intent
            confidence: medium
          - family: has_text
            intent: "The flagged row identifies the am3-positive decoy accession."
            expected_value: "E_am3_ctrl"
            tolerance: null
            element_identifier: null
            evidence: intent
            confidence: medium
          - family: has_text
            intent: "The flagged row's stop_codons column records position 7, matching the paper's Gene E codon-7 TGG->TAG (gpE_W7*) diagnostic mutation at genome position nt587 G->A."
            expected_value: "7"
            tolerance: null
            element_identifier: null
            evidence: intent
            confidence: medium

unresolved:
  - kind: fixture
    description: >-
      No small real Logan-shard kmindex index or Logan-derived LexicMap domain index exists for
      any of GENOMIC_PHG/GENOMIC_VRL/METAGENOMIC_ENV or Viral/BacteriaMetagenomic;
      kmindex_db_selection and lexicmap_index_selection therefore cannot be bound to fixtures
      carrying genuine phiX174 containment content at any tier smaller than usegalaxy.org's
      production Logan indices, in either test case.
    blocking: true
    suggested_resolution: >-
      Either obtain usegalaxy.org history/API access to run a real (if slow) end-to-end pass, or
      build a tiny local kmindex/LexicMap index (via kmindex_query's kmindex_build wrapper /
      lexicmap-index.xml) seeded with cds/E.fasta, cds/J.fasta plus the 3 decoy accessions
      described in test case gene_e_j_am3_diagnostic_synthetic_lexicmap_index, named to satisfy
      db_opts|kmindex / db_opts|lexicmap_index's data-table lookup.
  - kind: fixture
    description: >-
      The synthetic 3-decoy LexicMap index (wildtype-clean Gene E, am3-positive Gene E, clean
      Gene J) that test case gene_e_j_am3_diagnostic_synthetic_lexicmap_index depends on does not
      yet exist as a buildable artifact; this plan only specifies its required composition and
      the accession identifiers (E_wildtype_ctrl, E_am3_ctrl, J_clean_ctrl) its assertions
      reference.
    blocking: true
    suggested_resolution: >-
      implement-galaxy-workflow-test (or a follow-on find-test-data pass) should construct the 3
      decoy sequences (derived from real cds/E.fasta / cds/J.fasta with the am3 decoy carrying
      the exact nt587 G->A substitution) and build/register a tiny LexicMap index from them, or
      hand-construct the equivalent LexicMap hit-table TSV directly (per
      test-data-refs.json's confirmed lexicmap_search column schema) if index construction
      proves impractical.
  - kind: fixture
    description: >-
      Whether kmindex's test-only "register" value and lexicmap's db.lmi+db2.lmi test-data names
      (used in test case kmindex_wiring_smoke_generic_fixtures) are addressable from a
      workflow-level Planemo test job the same way they are from each tool's own functional test
      harness is unconfirmed.
    blocking: false
    suggested_resolution: >-
      Confirm via a trial planemo workflow_test invocation; the plan's existing existence-only
      assertion design for that test case already tolerates either outcome.
  - kind: fixture
    description: >-
      No verified phiX174-negative-control SRA accession is documented anywhere in the source
      project (test-data-refs.json gap "no-verified-phix174-negative-control-accession"); a true
      negative-control test case is therefore out of scope for this plan rather than fabricated.
    blocking: false
    suggested_resolution: >-
      User-supplied: name a specific SRA accession known/expected to have zero phiX174
      gene-panel containment, for use as a true-negative case in a future revision of this plan.
  - kind: output-label
    description: >-
      This skill's own declared inputs are the earlier template-era interface/data-flow briefs,
      but all workflow_label values in this plan were instead taken from the concrete
      gxformat2 draft (galaxy-workflow.gxwf.yml), per this run's explicit grounding instruction.
      A strict re-check of every label against the live draft (and against the workflow-label
      cross-check implement-galaxy-workflow-test already performs) is still worthwhile before
      finalizing, since this plan itself was not produced by that cross-check tool.
    blocking: false
    suggested_resolution: null

omissions:
  - target: "Full production-scale run (10 genes x 109 kmindex shards x up to 25 LexicMap indices, ~2.1M accessions)"
    reason: >-
      Explicitly out of scope for a fast workflow test. The real, already-computed full-scale
      numbers in test-data-refs.json
      (expected_output_ground_truth_full_scale_only: gene E total_accessions_evaluated=2392457,
      flagged_internal_stops=2215172, am3_stop_fraction_pct=92.59; gene J
      total_accessions_evaluated=2278534, flagged_internal_stops=63) are retained only as
      directional/sanity-bound references for a future full-scale validation pass against
      usegalaxy.org, not as exact-match assertions in this plan.
    category: out-of-scope
  - target: "kmindex_accession_union content assertions, and all lexicmap/streamer/manifest/am3-audit outputs in test case kmindex_wiring_smoke_generic_fixtures"
    reason: >-
      No small real Logan-shard kmindex index or Logan-derived LexicMap index exists; content in
      that test case has no phiX174 biological relationship, so only existence-level checks are
      asserted there. The content-bearing assertions live in the sibling test case instead.
    category: no-stable-checkpoint
  - target: "True negative-control test case (a confirmed phiX174-negative SRA accession)"
    reason: >-
      No accession is documented anywhere in the source project as a confirmed phiX174 non-hit;
      fabricating one would violate the no-fabrication rule this plan inherits from
      find-test-data (test-data-refs.json gap "no-verified-phix174-negative-control-accession").
    category: out-of-scope
  - target: "LexicMap raw per-gene hit table (lexicmap_search's out_file port)"
    reason: >-
      Not a promoted top-level workflow output (absent from galaxy-workflow.gxwf.yml's outputs:
      block); per galaxy-workflow-testability-design, only labeled workflow-level outputs are
      assertable by a Galaxy workflow test. This intermediate is exercised only indirectly via
      lexicmap_streamer_tiling_qc's own outputs.
    category: no-stable-checkpoint

warnings:
  - code: workflow-label-source-draft-not-brief
    message: >-
      This plan's declared inputs are the earlier template-era interface/data-flow briefs, but a
      concrete gxformat2 workflow draft (galaxy-workflow.gxwf.yml) and resolved test-data-refs.json
      were additionally available and used for grounding, per this run's explicit instructions.
      All workflow_label values were taken from the concrete draft's actual input/output ids, not
      re-derived from the brief's earlier proposed labels, which differ in places (e.g. the
      brief's prose "Gene E am3 quarantine audit" vs. the draft's actual id
      gene_e_am3_quarantine_audit).
    path: "workflow.label_source"
  - code: missing-fixture
    message: >-
      Two shared job inputs (kmindex_db_selection's and lexicmap_index_selection's backing
      indices) and one entire synthetic fixture (the 3-decoy LexicMap index) have no concrete,
      buildable location yet; see unresolved[0] and unresolved[1].
    path: "test_cases[*].job_inputs"
  - code: expression-derived-shape
    message: >-
      flatten_gene_summary_json_to_row's summary_row TSV column list (used for the
      ingestion-manifest and per-gene-summary assertions) was taken from
      galaxy-workflow.gxwf.yml's own step doc text, which itself cites the vendored
      lexicmap_streamer.py source at a pinned commit; this plan's assertion_intent values are
      best-effort against that documented contract, not against an actual executed run.
    path: "test_cases[1].expected_outputs"

GalaxyWorkflow — 14 step(s), 0 still drafty, 17 input(s), 8 output(s).

class: GalaxyWorkflow
id: phix174_sra_landscape_spikein_sieve
label: ΦX174 SRA Landscape & Spike-In Sieve (Workflow A)
doc: |
  Workflow A from the ΦX174 planetary-scale retrospective DMS pipeline (draft_manuscript.tex,
  Stages A, B, C, D2 only): combine the 10-gene CDS query panel, run a petabase k-mer
  containment screen (kmindex) across Logan SRA index shards, run a LexicMap streaming search
  per gene against Logan domain indices, run the custom LexicMapStreamer multi-HSP
  tiling/codon-QC/haplotype-collapsing pass, aggregate per-gene summaries into an ingestion
  manifest, and extract the Gene E am3 spike-in quarantine audit.

  Scope and provenance: freeform-summary.md (Stages A/B/C/D2), freeform-galaxy-interface.md
  (Workflow A section), freeform-galaxy-data-flow.md (nodes N1-N7), iwc-comparison-notes.md,
  open-requirements.ledger.yml. Workflows B/C/D from the interface brief are out of scope here.

  Two source-brief inputs are intentionally NOT declared as workflow inputs, per topology
  review in this Mold (see open-requirements ledger for citations):
  - `Reference genome (NC_001422.1)` — no node in this settled N1-N7 topology consumes the
    whole-genome FASTA; only the per-gene CDS FASTAs (gene_query_panel) are used, matching the
    source's own language about the "canonical gene coordinate frame."
  - `Nominal-taxonomy BioProject classification table` — a manual/registry-based curation
    artifact wired to no computational node; kept as external documentation/provenance, not a
    workflow input.
inputs:
  - id: gene_query_panel
    type: collection
    collection_type: sample_sheet
    format:
      - fasta
    doc: |
      Per-gene CDS query panel, 10 elements (element_identifier = gene symbol: A, B, C, D, E,
      F, G, H, J, K). Column definitions carry the per-gene LexicMap sensitivity overrides used
      for genes A/C/E/K (freeform-summary.md Stage C); populated only for those four genes, left
      unset (falling back to workflow-default LexicMap parameters) for the other six.
    column_definitions:
      - name: align_min_match_pident
        type: float
        optional: true
        description: Per-gene LexicMap align_min_match_pident override (~60.0 for A/C/E/K); unset for other genes.
      - name: align_min_match_len
        type: int
        optional: true
        description: Per-gene LexicMap align_min_match_len override (35-50 range for A/C/E/K); unset for other genes.
      - name: seed_min_prefix
        type: int
        optional: true
        description: Per-gene LexicMap seed_min_prefix override (15-17 range for A/C/E/K); unset for other genes.
      - name: min_qcov_per_genome
        type: float
        optional: true
        description: Per-gene LexicMap min_qcov_per_genome override (~30.0 for A/C/E/K); unset for other genes.
  - id: kmindex_zvalue
    type: integer
    optional: false
    default: 6
  - id: kmindex_threshold
    type: float
    optional: false
    default: 0.3
  - id: kmindex_output_format
    type: text
    optional: false
    default: json
  - id: kmindex_fast
    type: boolean
    optional: false
    default: false
  - id: lexicmap_top_n_genomes
    type: integer
    optional: false
    default: 0
  - id: lexicmap_advanced_all
    type: boolean
    optional: false
    default: true
  - id: tiling_qc_min_coverage
    type: float
    optional: false
    default: 0.8
  - id: tiling_qc_min_coverage_partial
    type: float
    optional: false
    default: 0.5
  - id: tiling_qc_min_pident
    type: float
    optional: false
    default: 60
  - id: tiling_qc_max_internal_stops
    type: integer
    optional: false
    default: 0
  - id: tiling_qc_sample_cap
    type: integer
    optional: false
    default: 10
  - id: tiling_qc_allow_frameshifts
    type: boolean
    optional: true
    doc: |
      No default asserted — production default of stream_lexicmap_msa.py's --allow-frameshifts
      flag is unconfirmed from prose alone. See open-requirements ledger entry
      `allow-frameshifts-default-unconfirmed`; expected to be resolved by reading the
      nekrut/disassembler repo during wrapper authoring (`lexicmapstreamer-wrapper-authoring-pending`).
  - id: gene_align_min_match_pident_panel
    type: collection
    collection_type: list
    doc: |
      Real per-gene LexicMap `align_min_match_pident` override, as a plain `list` collection
      whose element_identifiers match `gene_query_panel`'s (E/J for this test scope; A/C/E/K
      carry a real production override in the full 10-gene panel). Each element is a tiny
      text dataset holding that gene's resolved value: the real per-gene override (60.0 for E)
      where one exists, otherwise the pinned iuc/lexicmap/lexicmap_search 0.9.0+galaxy1
      wrapper's own declared tool default (70.0) verbatim. Added to fix a real defect: this
      value was previously carried as a `gene_query_panel` sample_sheet column and wired
      directly into lexicmap_search's scalar tool parameter, which cannot work because
      lexicmap_search has no sample_sheet awareness (see foundry-feedback.ledger.yml entry
      `lexicmap-search-sample-sheet-column-to-scalar-port-defect`). Consumed by
      `extract_align_min_match_pident_per_gene` (Galaxy core `param_value_from_file`), not
      lexicmap_search directly.
  - id: gene_align_min_match_len_panel
    type: collection
    collection_type: list
    doc: |
      Real per-gene LexicMap `align_min_match_len` override, same shape/rationale as
      `gene_align_min_match_pident_panel`: real override (35 for E) or the wrapper's own
      declared tool default (50) verbatim. Consumed by `extract_align_min_match_len_per_gene`.
  - id: gene_seed_min_prefix_panel
    type: collection
    collection_type: list
    doc: |
      Real per-gene LexicMap `seed_min_prefix` override, same shape/rationale as
      `gene_align_min_match_pident_panel`: real override (15 for E) or the wrapper's own
      declared tool default (17) verbatim. Consumed by `extract_seed_min_prefix_per_gene`.
  - id: gene_min_qcov_per_genome_panel
    type: collection
    collection_type: list
    doc: |
      Real per-gene LexicMap `min_qcov_per_genome` override, same shape/rationale as
      `gene_align_min_match_pident_panel`: real override (30.0 for E). Note: unlike the other
      three ports, this tool parameter is genuinely optional with no wrapper-declared default
      (unset means "no coverage floor applied"); since every element of this panel must carry
      a concrete value, genes without a real override use 0.0 as an explicit no-op floor
      (imposes no minimum), a deliberate approximation of "unset" documented here and in the
      feedback ledger, not a fabricated production value. Consumed by
      `extract_min_qcov_per_genome_per_gene`.
outputs:
  - id: kmindex_accession_union
    outputSource: kmindex_hit_dedup_max_score/accession_union
    type: data
    doc: Deduplicated, max-containment-score accession union across all queried kmindex shards.
  - id: clean_full_length_cds_haplotypes
    outputSource: lexicmap_streamer_tiling_qc/clean_msa_fasta
    type: collection
    doc: Per-gene clean full-length CDS haplotype MSA FASTA (sample_sheet, element_identifier = gene).
  - id: clean_haplotype_counts
    outputSource: lexicmap_streamer_tiling_qc/clean_haplotypes_tsv
    type: collection
    doc: Per-gene clean haplotype counts (weighted by observation count).
  - id: flagged_accessions
    outputSource: lexicmap_streamer_tiling_qc/flagged_accessions_fasta
    type: collection
    doc: Per-gene flagged (QC-failed) accession sequences.
  - id: flagged_audit_reasons
    outputSource: lexicmap_streamer_tiling_qc/flagged_tsv
    type: collection
    doc: Per-gene flagged-accession audit reasons table.
  - id: per_gene_ingestion_summary
    outputSource: lexicmap_streamer_tiling_qc/summary_json
    type: collection
    doc: Per-gene summary.json (quantile distributions, regime diagnostics, multi-HSP recovery counts) — source of the paper's Table 1 numbers.
  - id: ingestion_manifest_all_genes
    outputSource: join_gene_summary_rows_into_manifest/tabular_output
    type: data
    doc: lexicmap_9genes_ingestion_manifest.csv equivalent — all 10 per-gene summaries joined into one table.
  - id: gene_e_am3_quarantine_audit
    outputSource: extract_gene_e_am3_audit/output
    type: data
    doc: "Gene E am3 (nt587 G->A, codon-7 stop) spike-in quarantine diagnostic — the Gene E element of the per-gene flagged-accession audit table (flagged.tsv: accession, coverage, mean_pident, flag, n_stops, stop_codons, hsps_merged)."
comments:
  - type: frame
    label: Stage A — input assembly
    title: Stage A — input assembly
    color: blue
    position: [0, 0]
    size: [400, 300]
    contains_steps:
      - combine_gene_panel_to_bulk_fasta
  - type: frame
    label: Stage B — kmindex containment screen
    title: Stage B — kmindex containment screen
    color: green
    position: [450, 0]
    size: [400, 700]
    contains_steps:
      - kmindex_containment_screen
      - kmindex_hit_concat
      - kmindex_hit_dedup_max_score
  - type: frame
    label: Stage C — LexicMap search + LexicMapStreamer tiling/QC
    title: Stage C — LexicMap search + LexicMapStreamer tiling/QC
    color: orange
    position: [900, 0]
    size: [400, 500]
    contains_steps:
      - lexicmap_search
      - lexicmap_streamer_tiling_qc
  - type: frame
    label: Aggregation + Gene E am3 audit (Stage D2)
    title: Aggregation + Gene E am3 audit (Stage D2)
    color: pink
    position: [1350, 0]
    size: [400, 700]
    contains_steps:
      - flatten_gene_summary_json_to_row
      - join_gene_summary_rows_into_manifest
      - extract_gene_e_am3_audit
steps:
  - id: combine_gene_panel_to_bulk_fasta
    label: combine_gene_panel_to_bulk_fasta
    doc: |
      Combine per-gene panel into bulk query FASTA: plain concatenation of the 10-element
      sample_sheet's FASTA files into one multi-FASTA dataset (nml/collapse_collections/
      collapse_dataset v5.1.0, changeset 90981f86000f; confirmed via discover-shed-tool,
      single dominant hit). Per galaxy-collection-semantics.yml (SAMPLE_SHEET_MATCHES_LIST),
      a `sample_sheet` collection satisfies the wrapper's `list`-shaped multi-data
      `input_list` directly, no reshape needed. `one_header: false` and
      `filename.add_name: false` (tool defaults) preserve plain FASTA concatenation with no
      per-line headers or filename prepending. Output (port `output`, format_source:
      input_list) feeds kmindex_containment_screen's FASTA input.
    tool_id: toolshed.g2.bx.psu.edu/repos/nml/collapse_collections/collapse_dataset
    tool_version: 5.1.0
    tool_shed_repository:
      tool_shed: toolshed.g2.bx.psu.edu
      owner: nml
      name: collapse_collections
      changeset_revision: 90981f86000f
    in:
      input_list: gene_query_panel
    state:
      input_list:
        __class__: ConnectedValue
      one_header: false
      filename:
        add_name: false
    out:
      - output
  - id: kmindex_containment_screen
    label: kmindex_containment_screen
    doc: |
      kmindex containment screen (109 Logan shards) — iuc/kmindex/kmindex_query v0.6.1+galaxy4,
      changeset b6fa25b6b436; pin confirmed via discover-shed-tool and cross-verified against the
      changeset's upstream XML (tools-iuc 7681be7f40), because galaxy-tool-cache cannot parse this
      wrapper for any galaxy-suffixed version — see galaxy-tool-summary.json warnings[] and the
      feedback ledger.

      RESOLVED (2026-09-18, ledger `kmindex-lexicmap-index-selection-mechanism`): the 109 shard
      names are a LITERAL LIST in this step's state, not a delimited `text` workflow input. Tested
      live: the string "GENOMIC_PHG,GENOMIC_VRL" is REJECTED ("an invalid option ... was
      selected") because a `multiple: true` select reads the whole string as one option value,
      while the array ["GENOMIC_PHG","GENOMIC_VRL"] is accepted and fans out (job
      bbd44e69cb8906b5eadd64618b449770, history bbd44e69cb8906b57012ea23efccb56e: 32 GENOMIC_PHG +
      33 GENOMIC_VRL datasets). A gxformat2 `text` input cannot carry an array, so the former
      `kmindex_db_selection` input was removed rather than left silently unusable; changing the
      shard set now means editing this state list.

      `db_opts_selector: db` binds the wrapper's `kmindex` multi-select (options from the
      `kmindex` Tool Shed data table); one job takes the full selection and queries each index in
      turn (command template loops over `$db_opts.kmindex.fields.path.split(",")`), so this is not
      a Galaxy-level map-over axis. `threshold` has no wrapper-side default in this changeset
      (tools-iuc#8305 predates +galaxy4), so it must bind explicitly — it does, via
      `kmindex_threshold`. Output binds the wrapper's `output` collection (`list`,
      discover_datasets over `query_output/*.json`) — one JSON per shard. `output_matrix` never
      populates here (format is always json) and is correctly left unwired.
    tool_id: toolshed.g2.bx.psu.edu/repos/iuc/kmindex/kmindex_query
    tool_version: 0.6.1+galaxy4
    tool_shed_repository:
      tool_shed: toolshed.g2.bx.psu.edu
      owner: iuc
      name: kmindex
      changeset_revision: b6fa25b6b436
    in:
      fastx: combine_gene_panel_to_bulk_fasta/output
      zvalue: kmindex_zvalue
      threshold: kmindex_threshold
      format: kmindex_output_format
      fast: kmindex_fast
    state:
      fastx:
        __class__: ConnectedValue
      db_opts:
        db_opts_selector: db
        kmindex:
          - GENOMIC_BCT
          - GENOMIC_HUMAN
          - GENOMIC_INV
          - GENOMIC_MAM
          - GENOMIC_MICE
          - GENOMIC_PHG
          - GENOMIC_PLN
          - GENOMIC_PRI
          - GENOMIC_ROD
          - GENOMICSINGLECELL_BCT
          - GENOMICSINGLECELL_HUMAN
          - GENOMICSINGLECELL_INV
          - GENOMICSINGLECELL_MAM
          - GENOMICSINGLECELL_MICE
          - GENOMICSINGLECELL_PHG
          - GENOMICSINGLECELL_PLN
          - GENOMICSINGLECELL_PRI
          - GENOMICSINGLECELL_ROD
          - GENOMICSINGLECELL_UNKNOWN
          - GENOMICSINGLECELL_VRL
          - GENOMICSINGLECELL_VRT
          - GENOMIC_UNKNOWN
          - GENOMIC_VRL
          - GENOMIC_VRT
          - METAGENOMIC_BCT
          - METAGENOMIC_ENV
          - METAGENOMIC_HUMAN
          - METAGENOMIC_INV
          - METAGENOMIC_MAM
          - METAGENOMIC_MICE
          - METAGENOMIC_PHG
          - METAGENOMIC_PLN
          - METAGENOMIC_PRI
          - METAGENOMIC_ROD
          - METAGENOMIC_UNKNOWN
          - METAGENOMIC_VRL
          - METAGENOMIC_VRT
          - METATRANSCRIPTOMIC_BCT
          - METATRANSCRIPTOMIC_HUMAN
          - METATRANSCRIPTOMIC_INV
          - METATRANSCRIPTOMIC_MAM
          - METATRANSCRIPTOMIC_MICE
          - METATRANSCRIPTOMIC_PHG
          - METATRANSCRIPTOMIC_PLN
          - METATRANSCRIPTOMIC_PRI
          - METATRANSCRIPTOMIC_ROD
          - METATRANSCRIPTOMIC_UNKNOWN
          - METATRANSCRIPTOMIC_VRL
          - METATRANSCRIPTOMIC_VRT
          - OTHER_BCT
          - OTHER_HUMAN
          - OTHER_INV
          - OTHER_MAM
          - OTHER_MICE
          - OTHER_PHG
          - OTHER_PLN
          - OTHER_PRI
          - OTHER_ROD
          - OTHER_UNKNOWN
          - OTHER_VRL
          - OTHER_VRT
          - SYNTHETIC_BCT
          - SYNTHETIC_HUMAN
          - SYNTHETIC_INV
          - SYNTHETIC_MAM
          - SYNTHETIC_MICE
          - SYNTHETIC_PHG
          - SYNTHETIC_PLN
          - SYNTHETIC_PRI
          - SYNTHETIC_ROD
          - SYNTHETIC_UNKNOWN
          - SYNTHETIC_VRL
          - SYNTHETIC_VRT
          - TRANSCRIPTOMIC_BCT
          - TRANSCRIPTOMIC_HUMAN
          - TRANSCRIPTOMIC_INV
          - TRANSCRIPTOMIC_MAM
          - TRANSCRIPTOMIC_MICE
          - TRANSCRIPTOMIC_PHG
          - TRANSCRIPTOMIC_PLN
          - TRANSCRIPTOMIC_PRI
          - TRANSCRIPTOMIC_ROD
          - TRANSCRIPTOMICSINGLECELL_BCT
          - TRANSCRIPTOMICSINGLECELL_HUMAN
          - TRANSCRIPTOMICSINGLECELL_INV
          - TRANSCRIPTOMICSINGLECELL_MAM
          - TRANSCRIPTOMICSINGLECELL_MICE
          - TRANSCRIPTOMICSINGLECELL_PHG
          - TRANSCRIPTOMICSINGLECELL_PLN
          - TRANSCRIPTOMICSINGLECELL_PRI
          - TRANSCRIPTOMICSINGLECELL_ROD
          - TRANSCRIPTOMICSINGLECELL_UNKNOWN
          - TRANSCRIPTOMICSINGLECELL_VRL
          - TRANSCRIPTOMICSINGLECELL_VRT
          - TRANSCRIPTOMIC_UNKNOWN
          - TRANSCRIPTOMIC_VRL
          - TRANSCRIPTOMIC_VRT
          - VIRALRNA_BCT
          - VIRALRNA_HUMAN
          - VIRALRNA_INV
          - VIRALRNA_MAM
          - VIRALRNA_MICE
          - VIRALRNA_PHG
          - VIRALRNA_PLN
          - VIRALRNA_PRI
          - VIRALRNA_ROD
          - VIRALRNA_UNKNOWN
          - VIRALRNA_VRL
          - VIRALRNA_VRT
      zvalue:
        __class__: ConnectedValue
      threshold:
        __class__: ConnectedValue
      format:
        __class__: ConnectedValue
      fast:
        __class__: ConnectedValue
      verbose: error
    out:
      - output
  - id: kmindex_hit_concat
    label: kmindex_hit_concat
    doc: |
      Concatenate the 109-element per-shard kmindex hit JSON list collection into one dataset
      (nml/collapse_collections/collapse_dataset v5.1.0, changeset 90981f86000f; confirmed via
      discover-shed-tool, single dominant hit — same wrapper and pinned version already used by
      combine_gene_panel_to_bulk_fasta in this workflow). `input_list`'s `gx_data`/`multiple: true`
      shape accepts the `kmindex_containment_screen/output` list collection directly, no reshape
      needed. `one_header: false` and `filename.add_name: false` (tool defaults) preserve plain
      concatenation with no per-line headers or filename prepending, matching this wrapper's other
      use in the workflow. Output (port `output`, format_source: input_list) feeds
      kmindex_hit_dedup_max_score's dedup/max-score reduction — the built-in half of N3's
      "merge/union" node (iwc-comparison-notes.md: Collapse Collection handles concatenation; the
      dedup+max-score reduction is a separate custom step, see kmindex_hit_dedup_max_score below).
      This split exists because no single wrapper realizes both concatenation and keyed reduction
      (galaxy-workflow-draft-format.md's missing-combine-node rule).
    tool_id: toolshed.g2.bx.psu.edu/repos/nml/collapse_collections/collapse_dataset
    tool_version: 5.1.0
    tool_shed_repository:
      tool_shed: toolshed.g2.bx.psu.edu
      owner: nml
      name: collapse_collections
      changeset_revision: 90981f86000f
    in:
      input_list: kmindex_containment_screen/output
    state:
      input_list:
        __class__: ConnectedValue
      one_header: false
      filename:
        add_name: false
    out:
      - output
  - id: kmindex_hit_dedup_max_score
    label: kmindex_hit_dedup_max_score
    doc: |
      Deduplicate kmindex per-shard hit records by accession, keeping the maximum containment
      score per accession. Single-input pure transform reading kmindex_hit_concat's concatenated
      ~109-shard JSON hit maps, emitting the deduplicated accession union as one txt dataset
      (2,114,904 unique accessions per the paper).

      GalaxyUserTool authored from scratch (no Tool Shed wrapper does key-based dedup-with-max-
      score reduction over JSON; GNU datamash was considered and rejected since it requires
      already-tabular pre-sorted input, not JSON): `galaxy-user-tool-kmindex-hit-dedup-max-
      score.yml`, id `kmindex_hit_dedup_max_score` v1.0.0, container `python:3.13-slim`,
      pure-stdlib script. Handles both flat `{accession: score}` and per-query-nested
      `{query: {accession: score}}` kmindex output shapes via a recursive numeric-leaf walk over
      a repeated-JSON-document stream (Collapse Collection inserts no separator between shards).

      Full authoring/discovery provenance recorded in this Foundry run's open-requirements and
      feedback ledgers, not here, to stay within Galaxy's workflow doc size limit.
    tool_id: kmindex_hit_dedup_max_score
    tool_version: 1.0.0
    in:
      concatenated_hits: kmindex_hit_concat/output
    state:
      concatenated_hits:
        __class__: ConnectedValue
    out:
      - accession_union
  - id: extract_align_min_match_pident_per_gene
    label: extract_align_min_match_pident_per_gene
    doc: |
      Extract the real per-gene `align_min_match_pident` scalar from
      `gene_align_min_match_pident_panel`'s per-element text dataset (Galaxy core
      `param_value_from_file` v0.1.0, param_type float, output port `float_param`), mapped
      over the same E/J axis as gene_query_panel via matching element identifiers. Feeds
      lexicmap_search's `advanced_settings|align_min_match_pident` port. Added as the real
      fix for `lexicmap-search-sample-sheet-column-to-scalar-port-defect` (foundry-feedback
      ledger): lexicmap_search has no sample_sheet awareness, so a sample_sheet column can
      never be read into one of its scalar tool parameters directly — this per-parameter
      `list` collection + extraction-step pattern is the structurally correct replacement.
    tool_id: param_value_from_file
    tool_version: 0.1.0
    in:
      input1: gene_align_min_match_pident_panel
    state:
      input1:
        __class__: ConnectedValue
      param_type: float
      remove_newlines: true
    out:
      - float_param
  - id: extract_align_min_match_len_per_gene
    label: extract_align_min_match_len_per_gene
    doc: |
      Extract the real per-gene `align_min_match_len` scalar from
      `gene_align_min_match_len_panel` (Galaxy core `param_value_from_file` v0.1.0, param_type
      integer, output port `integer_param`), mapped over the E/J axis. Feeds lexicmap_search's
      `advanced_settings|align_min_match_len` port. Same rationale/fix as
      extract_align_min_match_pident_per_gene above.
    tool_id: param_value_from_file
    tool_version: 0.1.0
    in:
      input1: gene_align_min_match_len_panel
    state:
      input1:
        __class__: ConnectedValue
      param_type: integer
      remove_newlines: true
    out:
      - integer_param
  - id: extract_seed_min_prefix_per_gene
    label: extract_seed_min_prefix_per_gene
    doc: |
      Extract the real per-gene `seed_min_prefix` scalar from `gene_seed_min_prefix_panel`
      (Galaxy core `param_value_from_file` v0.1.0, param_type integer, output port
      `integer_param`), mapped over the E/J axis. Feeds lexicmap_search's
      `advanced_settings|seed_min_prefix` port. Same rationale/fix as
      extract_align_min_match_pident_per_gene above.
    tool_id: param_value_from_file
    tool_version: 0.1.0
    in:
      input1: gene_seed_min_prefix_panel
    state:
      input1:
        __class__: ConnectedValue
      param_type: integer
      remove_newlines: true
    out:
      - integer_param
  - id: extract_min_qcov_per_genome_per_gene
    label: extract_min_qcov_per_genome_per_gene
    doc: |
      Extract the real per-gene `min_qcov_per_genome` scalar from
      `gene_min_qcov_per_genome_panel` (Galaxy core `param_value_from_file` v0.1.0, param_type
      float, output port `float_param`), mapped over the E/J axis. Feeds lexicmap_search's
      `advanced_settings|min_qcov_per_genome` port. Same rationale/fix as
      extract_align_min_match_pident_per_gene above.
    tool_id: param_value_from_file
    tool_version: 0.1.0
    in:
      input1: gene_min_qcov_per_genome_panel
    state:
      input1:
        __class__: ConnectedValue
      param_type: float
      remove_newlines: true
    out:
      - float_param
  - id: nest_gene_panel_for_search
    label: nest_gene_panel_for_search
    doc: |
      Wrap each gene of `gene_query_panel` in its own single-element inner list, turning the flat
      `list` into a `list:list` of singletons (Galaxy core `__APPLY_RULES__`; the two
      `add_column_metadata: identifier0` rules give `list_identifiers` two columns to nest on, so
      outer and inner identifiers are both the gene name).

      WHY (2026-09-18, `lexicmap-search-query-port-reduces-not-maps`): lexicmap_search's `query`
      port is `multiple: true`, so a flat list connected to it is REDUCED -- one job receives all
      11 genes. Mapping then came only from the four parameter axes, so every job searched the
      whole panel under one gene's overrides: 11 jobs x 11 genes = 121 gene-searches for 11 genes'
      worth of result, and 37.0 GB of hit tables of which 3.4 GB was ever used (91% waste, with 7
      of the 11 outputs byte-identical). Feeding a `list:list` instead makes Galaxy map the outer
      level and reduce each inner list, so each job searches exactly one gene.

      Verified live on usegalaxy.org (invocation 38d6297a027f04c2): 3 genes -> 3 jobs, one query
      each, each carrying its own align_min_match_pident (C=60, K=65, J=70), output a flat `list`
      keyed by gene. Note the direct `/api/tools` path instead FLATTENS a list:list into a single
      job -- only the workflow path maps -- so this cannot be checked by running the tool ad hoc.
    tool_id: __APPLY_RULES__
    state:
      input:
        __class__: ConnectedValue
      rules:
        rules:
          - type: add_column_metadata
            value: identifier0
          - type: add_column_metadata
            value: identifier0
        mapping:
          - type: list_identifiers
            columns: [0, 1]
    in:
      input: gene_query_panel
    out:
      - output
  - id: lexicmap_search
    label: lexicmap_search
    doc: |
      LexicMap streaming search (iuc/lexicmap/lexicmap_search v0.9.0+galaxy1, changeset
      bcb6caec41eb; pin confirmed via discover-shed-tool). `db_opts_selector: db` binds the
      wrapper's `lexicmap_index` multi-select in-tool-call, not a Galaxy map-over axis. All 25
      Logan index categories are a LITERAL LIST in this step's state: like `db_opts|kmindex` it is
      a `multiple: true` select, rejecting a delimited string and requiring an array (ledger
      `kmindex-lexicmap-index-selection-mechanism`), so the `lexicmap_index_selection` text input
      was removed. All 25 verified against the live option set.

      CORRECTION (2026-09-18): previously the 5-name TARGETED_PHAGE_INDICES subset, per
      freeform-summary.md Stage C, which is wrong about what produced the paper. Both production
      submitters pass `ALL_LEXICMAP` (25): submit_all_structural_logan.py:66 (B/D/F/G/H/J) and
      submit_remaining_logan.py:101 (A/C/E/K). TARGETED_PHAGE_INDICES is only a default in
      pull_logan_alignments.py (which has --all-indices) and never produced Table 1. At 5 indices
      gene C evaluated 851,229 accessions vs the paper's 2,319,761. The four per-gene sensitivity
      overrides are each fed by an `extract_<name>_per_gene` step (`param_value_from_file` over a
      parallel `gene_<name>_panel` list whose element identifiers match `gene_query_panel`'s);
      ledger entries `lexicmap-search-sample-sheet-column-to-scalar-port-defect` and
      `lexicmap-search-flatten-shape-mismatch` cover the prior wiring and output-shape defects.

      `query` is now fed the `list:list` from `nest_gene_panel_for_search`, NOT `gene_query_panel`
      directly: the port is `multiple: true`, so a flat list is reduced into one job. See that
      step's doc for the full evidence. Each job now searches exactly one gene under that gene's
      own overrides, and the output is a flat `list` keyed by gene -- the shape
      lexicmap_streamer_tiling_qc already consumes, so nothing downstream changed. This cut
      121 gene-searches to 11 (~6.9x less LexicMap compute) and ~37 GB of hit tables to ~3.4 GB.
      Evidence: galaxy-vs-paper-comparison.md.
    tool_id: toolshed.g2.bx.psu.edu/repos/iuc/lexicmap/lexicmap_search
    tool_version: 0.9.0+galaxy1
    tool_shed_repository:
      tool_shed: toolshed.g2.bx.psu.edu
      owner: iuc
      name: lexicmap
      changeset_revision: bcb6caec41eb
    in:
      query: nest_gene_panel_for_search/output
      top_n_genomes: lexicmap_top_n_genomes
      advanced_settings|all: lexicmap_advanced_all
      advanced_settings|align_min_match_pident: extract_align_min_match_pident_per_gene/float_param
      advanced_settings|align_min_match_len: extract_align_min_match_len_per_gene/integer_param
      advanced_settings|seed_min_prefix: extract_seed_min_prefix_per_gene/integer_param
      advanced_settings|min_qcov_per_genome: extract_min_qcov_per_genome_per_gene/float_param
    state:
      query:
        __class__: ConnectedValue
      db_opts:
        db_opts_selector: db
        lexicmap_index:
          - BacteriaGenomic
          - BacteriaMetagenomic
          - BacteriaMetatranscriptomic
          - BacteriaTranscriptomic
          - FungiPlantGenomic
          - FungiPlantMetagenomic
          - FungiPlantMetatranscriptomic
          - FungiPlantTranscriptomic
          - HumanGenomic
          - HumanMetagenomic
          - HumanMetatranscriptomic
          - HumanTranscriptomic
          - MouseGenomic
          - MouseMetagenomic
          - MouseMetatranscriptomic
          - MouseTranscriptomic
          - OtherGenomic
          - Other
          - OtherMetagenomic
          - OtherMetatranscriptomic
          - OtherTranscriptomic
          - Singlecell
          - SinglecellRNA
          - Synthetic
          - Viral
      top_n_genomes:
        __class__: ConnectedValue
      advanced_settings:
        align_band: 100
        align_ext_len: 1000
        align_max_gap: 20
        align_min_match_len:
          __class__: ConnectedValue
        align_min_match_pident:
          __class__: ConnectedValue
        all:
          __class__: ConnectedValue
        load_whole_seeds: false
        max_evalue: 10
        max_query_conc: 12
        min_qcov_per_genome:
          __class__: ConnectedValue
        min_qcov_per_hsp: null
        seed_max_dist: 1000
        seed_max_gap: 1000
        seed_min_prefix:
          __class__: ConnectedValue
        seed_min_single_prefix: 19
        top_n_chains: 10
    out:
      - out_file
  - id: lexicmap_streamer_tiling_qc
    label: lexicmap_streamer_tiling_qc
    doc: |
      LexicMapStreamer: multi-HSP tiling, codon QC, and haplotype collapsing on LexicMap
      search hits. Takes the per-gene reference CDS FASTA (`gene_query_panel`) plus that gene's
      hit table (`lexicmap_search/out_file`) and emits 8 outputs. Wraps the private
      implementation at github.com/nekrut/disassembler (python/lexicmap_streamer.py, MIT, pure
      Python 3 stdlib), vendored as a GalaxyUserTool (`galaxy-user-tool.yml`, id
      `lexicmap_streamer` v1.0.2). `--allow-frameshifts` defaults OFF per the script's argparse;
      `--no-a2m` is passed since this workflow has no A2M port.

      CORRECTION (v1.0.0 -> v1.0.1): the script assumed each accession's HSP rows arrive
      contiguously. Real output is ranked globally, so rows split and reappear -- this crashed on
      real data and misclassified most split-HSP accessions beforehand. Fixed by buffering rows
      per accession and finalizing once, after full input consumption.

      CORRECTION (v1.0.1 -> v1.0.2, confirmed on history bbd44e69cb8906b54be4b938868d5a9d): the
      script never read the hit table's `query` column, so given a table covering several query
      genes -- which is exactly what `lexicmap_search` produces above -- it tiled every gene's
      HSPs onto whichever reference it was handed. All 11 per-gene cohorts had Gene A, truncated
      to the target length, as rank-1 haplotype (27-33%, ~100% reported pident): entropy inflated
      ~4x, non-E flagged stops 25-70x too high, Gene E am3 `gpE_W7*` suppressed from 92.59% to
      6.31%. Fixed by selecting only rows whose `query` equals the reference record id; a table
      with rows but none matching now raises. Verified: fixture unchanged (1 clean WT + 1 am3 at
      codon 7); a real 260 MB slice with the Gene A reference gives WT fraction 0.740 and 0.05%
      flagged stops -- the paper's Gene A regime (0.825, 0.05%) -- vs 0.307 and 7.8% when broken.
      Authoring provenance is in the open-requirements/feedback ledgers.
    tool_id: lexicmap_streamer
    tool_version: 1.0.2
    in:
      reference_fasta: gene_query_panel
      lexicmap_results: lexicmap_search/out_file
      min_coverage: tiling_qc_min_coverage
      min_coverage_partial: tiling_qc_min_coverage_partial
      min_pident: tiling_qc_min_pident
      max_internal_stops: tiling_qc_max_internal_stops
      sample_cap: tiling_qc_sample_cap
      allow_frameshifts: tiling_qc_allow_frameshifts
    state:
      reference_fasta:
        __class__: ConnectedValue
      lexicmap_results:
        __class__: ConnectedValue
      min_coverage:
        __class__: ConnectedValue
      min_coverage_partial:
        __class__: ConnectedValue
      min_pident:
        __class__: ConnectedValue
      max_internal_stops:
        __class__: ConnectedValue
      sample_cap:
        __class__: ConnectedValue
      allow_frameshifts:
        __class__: ConnectedValue
    out:
      - clean_msa_fasta
      - clean_haplotypes_tsv
      - clean_accessions_fasta
      - clean_expanded_accessions_fasta
      - flagged_accessions_fasta
      - flagged_tsv
      - cohort_ledger_tsv
      - summary_json
  - id: flatten_gene_summary_json_to_row
    label: flatten_gene_summary_json_to_row
    doc: |
      Flatten per-gene summary.json into one manifest row: mapped one call per gene over
      lexicmap_streamer_tiling_qc's summary_json collection.

      GalaxyUserTool authored from scratch (`galaxy-user-tool-flatten-gene-summary.yml`, id
      `flatten_gene_summary_json_to_row` v1.0.0, container `python:3.13-slim`, pure stdlib).
      `iuc/jq` was considered and rejected: its filter/arguments params can't access the mapped
      input's Galaxy collection `element_identifier`, which would need an extra producer step
      just to recover the gene symbol. This tool instead reads `element_identifier` directly in
      its shell_command, writing a two-line TSV (header + one row) whose first column is the
      gene symbol, followed by summary.json's flat scalar fields (status, reference metadata,
      regime, streamed/evaluated counts, cohort breakdown, recovery metrics, coverage/pident
      quantiles, haplotype/entropy/wildtype stats, runtime) — read directly from the real
      vendored lexicmap_streamer.py, not reverse-engineered from prose. The nested
      `top_10_haplotypes` field is intentionally excluded (not a flat scalar).

      Full authoring/discovery provenance recorded in this Foundry run's open-requirements and
      feedback ledgers, not here, to stay within Galaxy's workflow doc size limit.
    tool_id: flatten_gene_summary_json_to_row
    tool_version: 1.0.0
    in:
      summary_json: lexicmap_streamer_tiling_qc/summary_json
    state:
      summary_json:
        __class__: ConnectedValue
    out:
      - summary_row
  - id: join_gene_summary_rows_into_manifest
    label: join_gene_summary_rows_into_manifest
    doc: |
      Join per-gene summary rows into ingestion manifest.

      Tool Shed wrapper `iuc/collection_column_join` v0.0.3 (changeset `3ddd99c7efee`),
      confirmed via discover-shed-tool (single dominant search hit). Input port
      `input_tabular` (gx_data, multiple=true) takes flatten_gene_summary_json_to_row's mapped-
      over `summary_row` output (one single-row-plus-header TSV per gene, first column `gene`).
      Bound `identifier_column: 1` (the gene-symbol column), `has_header: 1`,
      `old_col_in_header: true` (wrapper default), `fill_char: ""` (uniform single-row-per-gene
      join, no missing cells expected), `include_outputs: []` (optional side output not needed).
      Output port `tabular_output` is the single joined manifest table, wired to workflow output
      `ingestion_manifest_all_genes`.
    tool_id: toolshed.g2.bx.psu.edu/repos/iuc/collection_column_join/collection_column_join
    tool_version: 0.0.3
    tool_shed_repository:
      tool_shed: toolshed.g2.bx.psu.edu
      owner: iuc
      name: collection_column_join
      changeset_revision: 3ddd99c7efee
    in:
      input_tabular: flatten_gene_summary_json_to_row/summary_row
    state:
      input_tabular:
        __class__: ConnectedValue
      identifier_column: 1
      has_header: 1
      old_col_in_header: true
      fill_char: ""
      include_outputs: []
    out:
      - tabular_output
  - id: extract_gene_e_am3_audit
    label: extract_gene_e_am3_audit
    doc: |
      Extract Gene E am3 quarantine audit: element_identifier == "E" from N5's per-gene
      `flagged_tsv` collection (toolshed built-in `__EXTRACT_DATASET__` 1.0.2 — a bare/stock id,
      no Tool Shed discovery or changeset pin applies).

      Reads `flagged_tsv`, not `summary_json`: summary.json's only per-haplotype detail
      (`top_10_haplotypes`) is built exclusively from the CLEAN cohort, and an am3 accession
      (a premature-stop mutant) is by construction routed to the FLAGGED cohort instead — so
      summary.json carries only an aggregate flagged count, no per-accession record. flagged.tsv
      carries one row per flagged accession including `stop_codons`, the actual codon position
      of the internal stop (e.g. "7" for the paper's codon-7 stop) — real per-accession,
      locus-specific detail already filtered to the anomalous cohort. Neither flagged.tsv nor
      cohort_ledger.tsv records the literal nucleotide substitution (only the codon index), so
      exact "nt587 G→A" reconstruction still needs the reference sequence alongside this table —
      a residual, non-blocking limitation.

      Full authoring/discovery provenance (including the correction from an earlier
      summary_json-based plan) recorded in this Foundry run's open-requirements and feedback
      ledgers, not here, to stay within Galaxy's workflow doc size limit. This is a same-step
      port choice (N7 reads N5, one of N5's 8 declared output ports), not a topology repair —
      no producer node was inserted or removed.
    tool_id: __EXTRACT_DATASET__
    tool_version: 1.0.2
    in:
      input: lexicmap_streamer_tiling_qc/flagged_tsv
    state:
      input:
        __class__: ConnectedValue
      which:
        which_dataset: by_identifier
        identifier: E
    out:
      - output

GalaxyWorkflowDraft — 9 step(s), 0 still drafty, 15 input(s), 8 output(s).

class: GalaxyWorkflowDraft
id: phix174_sra_landscape_spikein_sieve
label: "ΦX174 SRA Landscape & Spike-In Sieve (Workflow A)"
doc: |
  Workflow A from the ΦX174 planetary-scale retrospective DMS pipeline (draft_manuscript.tex,
  Stages A, B, C, D2 only): combine the 10-gene CDS query panel, run a petabase k-mer
  containment screen (kmindex) across Logan SRA index shards, run a LexicMap streaming search
  per gene against Logan domain indices, run the custom LexicMapStreamer multi-HSP
  tiling/codon-QC/haplotype-collapsing pass, aggregate per-gene summaries into an ingestion
  manifest, and extract the Gene E am3 spike-in quarantine audit.

  Scope and provenance: freeform-summary.md (Stages A/B/C/D2), freeform-galaxy-interface.md
  (Workflow A section), freeform-galaxy-data-flow.md (nodes N1-N7), iwc-comparison-notes.md,
  open-requirements.ledger.yml. Workflows B/C/D from the interface brief are out of scope here.

  Two source-brief inputs are intentionally NOT declared as workflow inputs, per topology
  review in this Mold (see open-requirements ledger for citations):
  - `Reference genome (NC_001422.1)` — no node in this settled N1-N7 topology consumes the
    whole-genome FASTA; only the per-gene CDS FASTAs (gene_query_panel) are used, matching the
    source's own language about the "canonical gene coordinate frame."
  - `Nominal-taxonomy BioProject classification table` — a manual/registry-based curation
    artifact wired to no computational node; kept as external documentation/provenance, not a
    workflow input.

inputs:
  - id: gene_query_panel
    type: collection
    collection_type: sample_sheet
    format: [fasta]
    doc: |
      Per-gene CDS query panel, 10 elements (element_identifier = gene symbol: A, B, C, D, E,
      F, G, H, J, K). Column definitions carry the per-gene LexicMap sensitivity overrides used
      for genes A/C/E/K (freeform-summary.md Stage C); populated only for those four genes, left
      unset (falling back to workflow-default LexicMap parameters) for the other six.
    column_definitions:
      - name: align_min_match_pident
        type: float
        optional: true
        description: "Per-gene LexicMap align_min_match_pident override (~60.0 for A/C/E/K); unset for other genes."
      - name: align_min_match_len
        type: int
        optional: true
        description: "Per-gene LexicMap align_min_match_len override (35-50 range for A/C/E/K); unset for other genes."
      - name: seed_min_prefix
        type: int
        optional: true
        description: "Per-gene LexicMap seed_min_prefix override (15-17 range for A/C/E/K); unset for other genes."
      - name: min_qcov_per_genome
        type: float
        optional: true
        description: "Per-gene LexicMap min_qcov_per_genome override (~30.0 for A/C/E/K); unset for other genes."

  - id: kmindex_db_selection
    type: text
    optional: false
    default: "GENOMIC_BCT,GENOMIC_HUMAN,GENOMIC_VRL,METAGENOMIC_ENV,METATRANSCRIPTOMIC_*,SYNTHETIC_*,VIRALRNA_*,TRANSCRIPTOMICSINGLECELL_*,...(109 Logan shard names, ALL_KMINDEX_DBS)"
    doc: |
      Multi-value selection over the 109 hardcoded Logan k-mer index shard names
      (freeform-summary.md Stage B, `ALL_KMINDEX_DBS`). Modeled as a single delimited string
      parameter consumed entirely inside the kmindex_query tool call (the tool's own multi-index
      query surface produces one JSON output per selected shard) rather than as a Galaxy
      workflow-level collection map-over axis. The wrapper-side widget is now confirmed (direct
      XML inspection of the pinned iuc/kmindex/kmindex_query 0.6.1+galaxy4 changeset): a native
      Galaxy `select` with `multiple="true"` (`db_opts|kmindex`, options from the `kmindex` Tool
      Shed data table) — not a repeat, not a per-element data input. Whether this scalar `text`
      workflow input's delimited-string value is accepted directly by that multi-select port at
      runtime, or needs adaptation to a literal array, remains open and non-blocking — see
      open-requirements ledger entry `kmindex-lexicmap-index-selection-mechanism`.

  - id: kmindex_zvalue
    type: integer
    optional: false
    default: 6
  - id: kmindex_threshold
    type: float
    optional: false
    default: 0.3
  - id: kmindex_output_format
    type: text
    optional: false
    default: "json"
  - id: kmindex_fast
    type: boolean
    optional: false
    default: false

  - id: lexicmap_index_selection
    type: text
    optional: false
    default: "Viral,BacteriaMetagenomic,HumanMetagenomic,OtherMetagenomic,Synthetic"
    doc: |
      Multi-value selection over LexicMap domain indices — defaults to the 5-index
      `TARGETED_PHAGE_INDICES` subset used for per-gene runs (freeform-summary.md Stage C); the
      full 25-index set is the alternative. Modeled the same way as `kmindex_db_selection`: a
      delimited string consumed inside the lexicmap_search tool call, whose own per-index output
      declaration produces the inner axis of the gene x index nested collection (see N4/lexicmap
      _search below). Same open mechanism question as kmindex — ledger
      `kmindex-lexicmap-index-selection-mechanism`.
  - id: lexicmap_top_n_genomes
    type: integer
    optional: false
    default: 0
  - id: lexicmap_advanced_all
    type: boolean
    optional: false
    default: true

  - id: tiling_qc_min_coverage
    type: float
    optional: false
    default: 0.80
  - id: tiling_qc_min_coverage_partial
    type: float
    optional: false
    default: 0.50
  - id: tiling_qc_min_pident
    type: float
    optional: false
    default: 60.0
  - id: tiling_qc_max_internal_stops
    type: integer
    optional: false
    default: 0
  - id: tiling_qc_sample_cap
    type: integer
    optional: false
    default: 10
  - id: tiling_qc_allow_frameshifts
    type: boolean
    optional: true
    doc: |
      No default asserted — production default of stream_lexicmap_msa.py's --allow-frameshifts
      flag is unconfirmed from prose alone. See open-requirements ledger entry
      `allow-frameshifts-default-unconfirmed`; expected to be resolved by reading the
      nekrut/disassembler repo during wrapper authoring (`lexicmapstreamer-wrapper-authoring-pending`).

outputs:
  - id: kmindex_accession_union
    outputSource: kmindex_hit_dedup_max_score/accession_union
    type: data
    doc: "Deduplicated, max-containment-score accession union across all queried kmindex shards."
  - id: clean_full_length_cds_haplotypes
    outputSource: lexicmap_streamer_tiling_qc/clean_msa_fasta
    type: collection
    doc: "Per-gene clean full-length CDS haplotype MSA FASTA (sample_sheet, element_identifier = gene)."
  - id: clean_haplotype_counts
    outputSource: lexicmap_streamer_tiling_qc/clean_haplotypes_tsv
    type: collection
    doc: "Per-gene clean haplotype counts (weighted by observation count)."
  - id: flagged_accessions
    outputSource: lexicmap_streamer_tiling_qc/flagged_accessions_fasta
    type: collection
    doc: "Per-gene flagged (QC-failed) accession sequences."
  - id: flagged_audit_reasons
    outputSource: lexicmap_streamer_tiling_qc/flagged_tsv
    type: collection
    doc: "Per-gene flagged-accession audit reasons table."
  - id: per_gene_ingestion_summary
    outputSource: lexicmap_streamer_tiling_qc/summary_json
    type: collection
    doc: "Per-gene summary.json (quantile distributions, regime diagnostics, multi-HSP recovery counts) — source of the paper's Table 1 numbers."
  - id: ingestion_manifest_all_genes
    outputSource: join_gene_summary_rows_into_manifest/tabular_output
    type: data
    doc: "lexicmap_9genes_ingestion_manifest.csv equivalent — all 10 per-gene summaries joined into one table."
  - id: gene_e_am3_quarantine_audit
    outputSource: extract_gene_e_am3_audit/output
    type: data
    doc: "Gene E am3 (nt587 G->A, codon-7 stop) spike-in quarantine diagnostic — the Gene E element of the per-gene flagged-accession audit table (flagged.tsv: accession, coverage, mean_pident, flag, n_stops, stop_codons, hsps_merged)."

comments:
  - type: frame
    label: "Stage A — input assembly"
    title: "Stage A — input assembly"
    contains_steps: [combine_gene_panel_to_bulk_fasta]
  - type: frame
    label: "Stage B — kmindex containment screen"
    title: "Stage B — kmindex containment screen"
    contains_steps: [kmindex_containment_screen, kmindex_hit_concat, kmindex_hit_dedup_max_score]
  - type: frame
    label: "Stage C — LexicMap search + LexicMapStreamer tiling/QC"
    title: "Stage C — LexicMap search + LexicMapStreamer tiling/QC"
    contains_steps: [lexicmap_search, lexicmap_streamer_tiling_qc]
  - type: frame
    label: "Aggregation + Gene E am3 audit (Stage D2)"
    title: "Aggregation + Gene E am3 audit (Stage D2)"
    contains_steps: [flatten_gene_summary_json_to_row, join_gene_summary_rows_into_manifest, extract_gene_e_am3_audit]

steps:
  # ---- N1: fan-in / combine (Resolution: Identity-pinned) ----
  - id: combine_gene_panel_to_bulk_fasta
    label: combine_gene_panel_to_bulk_fasta
    doc: |
      Combine per-gene panel into bulk query FASTA: plain concatenation of the 10-element
      sample_sheet's FASTA files into one multi-FASTA dataset (nml/collapse_collections/
      collapse_dataset v5.1.0, changeset 90981f86000f; confirmed via discover-shed-tool,
      single dominant hit). Per galaxy-collection-semantics.yml (SAMPLE_SHEET_MATCHES_LIST),
      a `sample_sheet` collection satisfies the wrapper's `list`-shaped multi-data
      `input_list` directly, no reshape needed. `one_header: false` and
      `filename.add_name: false` (tool defaults) preserve plain FASTA concatenation with no
      per-line headers or filename prepending. Output (port `output`, format_source:
      input_list) feeds kmindex_containment_screen's FASTA input.
    tool_id: toolshed.g2.bx.psu.edu/repos/nml/collapse_collections/collapse_dataset
    tool_version: 5.1.0
    tool_shed_repository:
      tool_shed: toolshed.g2.bx.psu.edu
      owner: nml
      name: collapse_collections
      changeset_revision: 90981f86000f
    in:
      input_list: gene_query_panel
    state:
      input_list:
        __class__: ConnectedValue
      one_header: false
      filename:
        add_name: false
    out:
      - output

  # ---- N2: kmindex containment screen (Resolution: Identity-pinned) ----
  - id: kmindex_containment_screen
    label: kmindex_containment_screen
    doc: |
      kmindex containment screen (109 Logan shards) — iuc/kmindex/kmindex_query v0.6.1+galaxy4,
      changeset b6fa25b6b436; identity+version pin confirmed via discover-shed-tool (single
      dominant hit, no contradiction) and cross-verified directly against the pinned changeset's
      real upstream XML source (tools-iuc commit 7681be7f40, matching the Tool Shed's own
      remote_repository_url for this changeset) because galaxy-tool-cache add/summarize cannot
      parse this wrapper for any galaxy-suffixed version (0.6.0+galaxy1 through 0.6.1+galaxy5) —
      see galaxy-tool-summary.json warnings[] and the feedback ledger for the reproducible defect.

      `db_opts_selector: db` ("Locally installed kmindex indexes") binds `kmindex_db_selection`
      (delimited shard-name text input) to the wrapper's `kmindex` multi-select
      (`db_opts|kmindex`, native Galaxy `multiple: true`, options from the `kmindex` Tool Shed
      data table) — a single job accepts the full multi-value shard selection and internally
      queries each index in turn, matching the already-committed in-tool-call binding (ledger
      `kmindex-lexicmap-index-selection-mechanism`, now further confirmed for the kmindex side by
      direct XML inspection: the command template loops `#for $i, $INDEX in enumerate($INDICES)`
      over `$db_opts.kmindex.fields.path.split(",")`), not a second Galaxy-level map-over axis.

      `threshold` has no wrapper-side default in this changeset (tools-iuc#8305 "remove threshold
      default" predates +galaxy4); the param is `optional=false` with no `value=` attribute, so it
      must bind explicitly — which it does via `kmindex_threshold`.

      Output binds to the wrapper's `output` collection (`list`, discover_datasets over
      `query_output/*.json`, active when format!='matrix') — one JSON element per selected shard,
      matching _plan_out exactly. The sibling `output_matrix` collection never populates for this
      step (format is always "json" here) and is correctly left unwired.
    tool_id: toolshed.g2.bx.psu.edu/repos/iuc/kmindex/kmindex_query
    tool_version: 0.6.1+galaxy4
    tool_shed_repository:
      tool_shed: toolshed.g2.bx.psu.edu
      owner: iuc
      name: kmindex
      changeset_revision: b6fa25b6b436
    in:
      fastx: combine_gene_panel_to_bulk_fasta/output
      db_opts|kmindex: kmindex_db_selection
      zvalue: kmindex_zvalue
      threshold: kmindex_threshold
      format: kmindex_output_format
      fast: kmindex_fast
    state:
      fastx:
        __class__: ConnectedValue
      db_opts:
        db_opts_selector: db
        kmindex:
          __class__: ConnectedValue
      zvalue:
        __class__: ConnectedValue
      threshold:
        __class__: ConnectedValue
      format:
        __class__: ConnectedValue
      fast:
        __class__: ConnectedValue
      verbose: error
    out:
      - output

  # ---- N3a: fan-in concat half (Resolution: Identity-pinned, confirmed) ----
  - id: kmindex_hit_concat
    label: kmindex_hit_concat
    doc: |
      Concatenate the 109-element per-shard kmindex hit JSON list collection into one dataset
      (nml/collapse_collections/collapse_dataset v5.1.0, changeset 90981f86000f; confirmed via
      discover-shed-tool, single dominant hit — same wrapper and pinned version already used by
      combine_gene_panel_to_bulk_fasta in this workflow). `input_list`'s `gx_data`/`multiple: true`
      shape accepts the `kmindex_containment_screen/output` list collection directly, no reshape
      needed. `one_header: false` and `filename.add_name: false` (tool defaults) preserve plain
      concatenation with no per-line headers or filename prepending, matching this wrapper's other
      use in the workflow. Output (port `output`, format_source: input_list) feeds
      kmindex_hit_dedup_max_score's dedup/max-score reduction — the built-in half of N3's
      "merge/union" node (iwc-comparison-notes.md: Collapse Collection handles concatenation; the
      dedup+max-score reduction is a separate custom step, see kmindex_hit_dedup_max_score below).
      This split exists because no single wrapper realizes both concatenation and keyed reduction
      (galaxy-workflow-draft-format.md's missing-combine-node rule).
    tool_id: toolshed.g2.bx.psu.edu/repos/nml/collapse_collections/collapse_dataset
    tool_version: 5.1.0
    tool_shed_repository:
      tool_shed: toolshed.g2.bx.psu.edu
      owner: nml
      name: collapse_collections
      changeset_revision: 90981f86000f
    in:
      input_list: kmindex_containment_screen/output
    state:
      input_list:
        __class__: ConnectedValue
      one_header: false
      filename:
        add_name: false
    out:
      - output

  # ---- N3b: fan-in custom dedup/max-score (Resolution: Deferred) ----
  - id: kmindex_hit_dedup_max_score
    label: kmindex_hit_dedup_max_score
    doc: |
      Deduplicate kmindex per-shard hit records by accession, keeping the maximum containment
      score per accession (GalaxyUserTool, authored this iteration): single-input pure transform
      reading kmindex_hit_concat's concatenated ~109-shard JSON hit maps, emitting the
      deduplicated accession union as one txt dataset (2,114,904 unique accessions per the
      paper; freeform-summary.md Stage B/C', today implemented ad hoc as `harvest_collection()`
      in process_kmindex_with_disassembler.py).

      WRAPPER RESOLUTION (discover-shed-tool then author-galaxy-tool-wrapper, 2026-09-17):
      `gxwf tool-search` on "datamash", "sort", "unique", and "group by" surfaced no wrapper
      performing key-based dedup-with-max-score reduction over JSON input (see
      galaxy-tool-pin.json, status "miss"). The closest structural match, GNU datamash
      (iuc/datamash_ops, agordon/datamash_wrapper -- `groupby -g <col> max <col>`), was rejected:
      it requires already-tabular, pre-sorted-by-key delimited input and cannot parse JSON, so
      adopting it would only relocate the custom transform (a JSON-to-TSV flatten + sort)
      upstream rather than eliminate it. This resolves open-requirements ledger entry
      `kmindex-hit-merge-no-corpus-precedent` (Collapse Collection, the prior step, handles
      concatenation; this is the small custom reduction that ledger entry recommended). A
      purpose-built GalaxyUserTool was authored at
      `galaxy-user-tool-kmindex-hit-dedup-max-score.yml` -- id `kmindex_hit_dedup_max_score`,
      version `1.0.0`, container `python:3.13-slim`, pure-stdlib (json/argparse) script run via
      `python kmindex_hit_dedup_max_score.py`. Per the already-recorded provenance
      (`advance-galaxy-draft-step-udt-binding-undocumented`), this step's `tool_id`/`tool_version`
      mirror the UDT's own declared identity, matching the same reasoned convention already used
      for this run's other two authored UDTs (lexicmap_streamer, flatten_gene_summary_json_to_row).

      Input shape assumption carried into the UDT itself (documented there, not fabricated here):
      Collapse Collection concatenates the per-shard JSON files with no inserted separator, and
      kmindex's own per-shard JSON could nest per-query or be a flat accession->score map (its
      schema could not be fetched this run, ledger
      `tool-util-cli-toolshed-fetch-rejects-real-filtered-list-collection-output`) -- the script
      handles both via a recursive numeric-leaf walk over a repeated-`raw_decode` document stream.
    tool_id: kmindex_hit_dedup_max_score
    tool_version: "1.0.0"
    in:
      concatenated_hits: kmindex_hit_concat/output
    state:
      concatenated_hits:
        __class__: ConnectedValue
    out:
      - accession_union

  # ---- N4a: LexicMap streaming search, per-gene map-over (Resolution: Resolved) ----
  - id: lexicmap_search
    label: lexicmap_search
    doc: |
      LexicMap streaming search, mapped over gene_query_panel (sample_sheet, 10 elements) —
      one query call per gene (iuc/lexicmap/lexicmap_search v0.9.0+galaxy1, changeset
      bcb6caec41eb; identity+version pin confirmed via discover-shed-tool, single dominant
      hit, no contradiction). `db_opts_selector: db` ("Locally installed LexicMap indexes")
      binds `lexicmap_index_selection` (delimited index-name text input) to the wrapper's
      `lexicmap_index` multi-select — the same in-tool-call binding already committed for
      `kmindex_db_selection` (ledger `kmindex-lexicmap-index-selection-mechanism`), not a
      second Galaxy-level map-over axis. The four per-gene sensitivity overrides
      (align_min_match_pident, align_min_match_len, seed_min_prefix, min_qcov_per_genome)
      bind to `gene_query_panel`'s matching optional per-element columns, falling through to
      the wrapper's own defaults (70 / 50 / 17 / unset) when a gene's columns are unset —
      realizing the two-regime design (freeform-summary.md Stage C) without a separate
      pick_value step, per _plan_state's now-discharged idiom.

      CORRECTION vs. prior plan (see open-requirements ledger, new entry
      `lexicmap-search-flatten-shape-mismatch`): the pinned changeset's `parsed_tool` declares
      exactly one output, `out_file` (plain `data`, format `tabular`, no `discover_datasets`,
      no per-index collection output) — there is no "tool's own per-index output declaration"
      to supply an inner axis. Mapped over the single `gene_query_panel` axis, this step's
      real output shape is a flat `list` (one aggregate hits table per gene, covering all
      selected indices for that gene), not the `list:list` the template's _plan_out assumed.
      This refutes the specific "inner axis" claim recorded against
      `kmindex-lexicmap-index-selection-mechanism`; that entry's note has been updated
      in place rather than left standing uncorrected.
    tool_id: toolshed.g2.bx.psu.edu/repos/iuc/lexicmap/lexicmap_search
    tool_version: 0.9.0+galaxy1
    tool_shed_repository:
      tool_shed: toolshed.g2.bx.psu.edu
      owner: iuc
      name: lexicmap
      changeset_revision: bcb6caec41eb
    in:
      query: gene_query_panel
      db_opts|lexicmap_index: lexicmap_index_selection
      top_n_genomes: lexicmap_top_n_genomes
      advanced_settings|all: lexicmap_advanced_all
      advanced_settings|align_min_match_pident: gene_query_panel
      advanced_settings|align_min_match_len: gene_query_panel
      advanced_settings|seed_min_prefix: gene_query_panel
      advanced_settings|min_qcov_per_genome: gene_query_panel
    state:
      query:
        __class__: ConnectedValue
      db_opts:
        db_opts_selector: db
        lexicmap_index:
          __class__: ConnectedValue
      top_n_genomes:
        __class__: ConnectedValue
      advanced_settings:
        align_band: 100
        align_ext_len: 1000
        align_max_gap: 20
        align_min_match_len:
          __class__: ConnectedValue
        align_min_match_pident:
          __class__: ConnectedValue
        all:
          __class__: ConnectedValue
        load_whole_seeds: false
        max_evalue: 10
        max_query_conc: 12
        min_qcov_per_genome:
          __class__: ConnectedValue
        min_qcov_per_hsp: null
        seed_max_dist: 1000
        seed_max_gap: 1000
        seed_min_prefix:
          __class__: ConnectedValue
        seed_min_single_prefix: 19
        top_n_chains: 10
    out:
      - out_file

  # ---- N4b: index-axis reduce — REMOVED by repair-galaxy-draft-topology, see below ----

  # ---- N5: LexicMapStreamer, map-over gene, 8 parallel per-gene outputs (Resolution: Deferred) ----
  - id: lexicmap_streamer_tiling_qc
    label: lexicmap_streamer_tiling_qc
    doc: |
      LexicMapStreamer: multi-HSP tiling, codon QC, haplotype collapsing.

      TOPOLOGY REPAIR (repair-galaxy-draft-topology, 2026-09-17, closing ledger blocking
      entry `lexicmap-search-flatten-shape-mismatch`): the former N4b `flatten_lexicmap_
      results_by_gene` (`__FLATTEN__`, gene x index list:list -> list) has been removed.
      lexicmap_search's real, confirmed wrapper output (`out_file`, mapped over
      `gene_query_panel`) is already a flat `list` — one aggregate hits table per gene,
      covering all selected indices for that gene — with no index axis to collapse.
      `__FLATTEN__` had nothing to flatten and its declared output was therefore
      uncomputable as specified. This step's input is rewired directly onto
      `lexicmap_search/out_file`, which already carries the exact shape (list, 10 elements,
      gene-keyed) this step's own doc says it expects ("Mapped one call per gene over the
      flattened LexicMap results (list, 10 elements)") — no substitute producer was needed.

      WRAPPER RESOLUTION (author-galaxy-tool-wrapper, 2026-09-17): discover-shed-tool ran
      three query variants ("lexicmap streamer", "haplotype tiling qc", "disassembler")
      against the Tool Shed lexical index — zero hits on all three (see
      galaxy-tool-pin.json, status "miss"). Per the already-recorded provenance
      (`lexicmapstreamer-implementation-provenance`), this is not a from-scratch tool: the
      real implementation was cloned and read directly from its private source,
      https://github.com/nekrut/disassembler (MIT license, (c) Anton Nekrutenko; repo HEAD
      a3999eb60ae78a804d0200b4a1241038077b6e6a; python/lexicmap_streamer.py last touched at
      commit a51eb58d6c1ca941d3fb8d6adf5e8160c3926b91). Confirmed by direct read: the
      script is pure Python 3 standard library (os, sys, gzip, math, json, time, argparse,
      urllib.request, pathlib, collections) — no third-party imports; the repo's
      environment.yml conda pins (biopython, mafft, fasttree, blast, ...) are for the
      repo's *other* scripts (disassemble.py, build_walker_codon_msa.py, the Rust walker),
      not this one. A GalaxyUserTool wrapper vendoring the script verbatim (as a
      `configfiles` entry, run via `python lexicmap_streamer.py`) was authored at
      `galaxy-user-tool.yml` — id `lexicmap_streamer`, version `1.0.0`, container
      `python:3.13-slim`, `--no-a2m` passed since the settled 8-output shape has no A2M
      port. This discharges open-requirements ledger entry
      `lexicmapstreamer-wrapper-authoring-pending`, and separately resolves
      `allow-frameshifts-default-unconfirmed`: the script's argparse declares
      `--allow-frameshifts` as `action="store_true"` with no `default=True` — the
      confirmed production default is OFF/False.

      CORRECTION vs. prior `_plan_in` (superseded, kept below for provenance): reading the
      real script's argparse shows it requires TWO inputs, not one — the per-gene
      reference CDS FASTA (`-q/--ref`, required, used to project HSPs onto and to
      translate/QC) in addition to the LexicMap hit table (`-f`). The reference FASTA is
      wired here to `gene_query_panel`, the exact same per-gene collection already
      consumed by `lexicmap_search`'s own `query` port a few steps upstream (see that
      step's `in: query: gene_query_panel`) — not a new workflow input, and not a
      topology change (the mapped gene axis is unchanged), just a previously-missed
      second data port on this step.

      UNRESOLVED ASSUMPTION carried forward (see feedback ledger entry
      `advance-galaxy-draft-step-udt-binding-undocumented`): the packaged reference set
      (galaxy-workflow-draft-format notes, the galaxy-workflow-draft JSON Schema, and
      implement-galaxy-tool-step's own SKILL.md) documents how a step resolves to a Tool
      Shed pin (tool_id + tool_shed_repository + changeset) or a bare/stock built-in id,
      but nowhere documents the concrete step shape once author-galaxy-tool-wrapper
      produces a `GalaxyUserTool` — no `tool_shed_repository` block applies, and no
      convention is given for whether `tool_id`/`tool_version` should mirror the UDT's own
      `id`/`version` fields. This step's `tool_id: lexicmap_streamer` / `tool_version:
      "1.0.0"` is a reasoned choice (mirror the UDT's own declared identity, by analogy to
      how a bare/stock id like `Filter1` carries no tool_shed_repository either), not a
      documented contract. `gxwf draft-validate --concrete` accepted this shape without
      complaint (2 ok / 0 fail on tool state, this step among the "ok"), which is
      reassuring runtime evidence but still not a documented guarantee.
    tool_id: lexicmap_streamer
    tool_version: "1.0.0"
    in:
      reference_fasta: gene_query_panel
      lexicmap_results: lexicmap_search/out_file
      min_coverage: tiling_qc_min_coverage
      min_coverage_partial: tiling_qc_min_coverage_partial
      min_pident: tiling_qc_min_pident
      max_internal_stops: tiling_qc_max_internal_stops
      sample_cap: tiling_qc_sample_cap
      allow_frameshifts: tiling_qc_allow_frameshifts
    state:
      reference_fasta:
        __class__: ConnectedValue
      lexicmap_results:
        __class__: ConnectedValue
      min_coverage:
        __class__: ConnectedValue
      min_coverage_partial:
        __class__: ConnectedValue
      min_pident:
        __class__: ConnectedValue
      max_internal_stops:
        __class__: ConnectedValue
      sample_cap:
        __class__: ConnectedValue
      allow_frameshifts:
        __class__: ConnectedValue
    out:
      - clean_msa_fasta
      - clean_haplotypes_tsv
      - clean_accessions_fasta
      - clean_expanded_accessions_fasta
      - flagged_accessions_fasta
      - flagged_tsv
      - cohort_ledger_tsv
      - summary_json

  # ---- N6a: JSON-to-row flatten (Resolution: Resolved) ----
  - id: flatten_gene_summary_json_to_row
    label: flatten_gene_summary_json_to_row
    doc: |
      Flatten per-gene summary.json into one manifest row (GalaxyUserTool, authored this
      iteration): mapped one call per gene over lexicmap_streamer_tiling_qc's summary_json
      collection.

      WRAPPER RESOLUTION (discover-shed-tool then author-galaxy-tool-wrapper, 2026-09-17):
      `gxwf tool-search "json to tabular"` surfaced only generic JSON-transform tools, the
      closest being `iuc/jq` ("query and transform JSON documents"; confirmed via
      galaxy-tool-cache add/summarize, changeset 4e62e523c2b6, version 1.8.2+galaxy0 -- a real,
      installable Tool Shed wrapper with a `tsv_output` option that marks its output tabular).
      Rejected as the wrapper for this step: `iuc/jq`'s `filter`/`arguments` parameters are
      plain text/JSON values with no access to the mapped input dataset's Galaxy collection
      `element_identifier`; using it here would require inserting a separate
      `collection_element_identifiers`-style producer step and a second wired port purely to
      recover the gene symbol as a manifest column -- an avoidable topology change, not a
      genuine tool-shape gap. A small purpose-built GalaxyUserTool
      (`galaxy-user-tool-flatten-gene-summary.yml`, id `flatten_gene_summary_json_to_row`,
      version `1.0.0`, container `python:3.13-slim`, pure stdlib) reads the input dataset's
      `element_identifier` directly in its `shell_command`
      (`$(inputs.summary_json.element_identifier)`) -- confirmed valid against the installed
      `@galaxy-tool-util/schema` package's `gx-data.js` parameter schema, whose `job_runtime`
      state representation's File object declares `element_identifier: S.optional(S.String)`
      alongside `path`, the same idiom documented for classic Galaxy tool XML
      (`convert-nfcore-module-to-galaxy-tool` note: "Galaxy's `$input.element_identifier` is
      the right substitute"). No extra port or producer step needed.

      Discharges open-requirements ledger `n6-json-flatten-no-corpus-precedent`'s remaining
      Deferred obligation: the wrapper's own vendored script writes a two-line TSV (header +
      one data row) whose first column, `gene`, is this dataset's `element_identifier` --
      preserving it end-to-end into the row content itself (not just Galaxy metadata), matching
      join_gene_summary_rows_into_manifest's plan ("identifier_column pointing at the
      gene-symbol column"). Remaining columns mirror summary.json's flat scalar fields (status,
      reference_name/length/is_cds, regime + regime_warning, total_streamed_rows,
      total_accessions_evaluated, cohort_breakdown.*, recovery_metrics.* [including
      multi_hsp_rescued_accessions], coverage_quantiles.* / pident_quantiles.*
      [p10/p25/p50_median/p75/p90/mean], clean_unique_haplotypes, shannon_entropy_bits,
      wildtype_count, wildtype_fraction, runtime_seconds) -- read directly from this run's own
      vendored lexicmap_streamer.py (galaxy-user-tool.yml, `summary_data` construction,
      commit a51eb58d6c1ca941d3fb8d6adf5e8160c3926b91) rather than reverse-engineered from
      prose. `top_10_haplotypes` (a nested list) is intentionally excluded -- not a flat
      per-gene scalar.

      `gxwf validate-tool-source galaxy-user-tool-flatten-gene-summary.yml` returns `OK`.
    tool_id: flatten_gene_summary_json_to_row
    tool_version: "1.0.0"
    in:
      summary_json: lexicmap_streamer_tiling_qc/summary_json
    state:
      summary_json:
        __class__: ConnectedValue
    out:
      - summary_row

  # ---- N6b: tabular join/concat (Resolution: Identity-pinned) ----
  - id: join_gene_summary_rows_into_manifest
    label: join_gene_summary_rows_into_manifest
    doc: |
      Join per-gene summary rows into ingestion manifest.

      WRAPPER RESOLUTION (advance-galaxy-draft-step, discover-shed-tool discovery-confirm,
      2026-09-17): the phase-5 identity pin (iuc/collection_column_join, tool_id
      `collection_column_join`, version 0.0.3) was confirmed rather than re-derived. Query
      `column join` against the Tool Shed lexical index (the underscored token
      `collection_column_join` was not queried verbatim per component-tool-shed-search's
      normalization guidance) returns iuc/collection_column_join ("Column join" / "on multiple
      datasets") as the single dominant hit (score 44.85), tied on score with an mvdbeek fork of
      the identical tool id/name -- iuc retained as the maintained/canonical owner, matching the
      pin. `gxwf tool-versions` lists 0.0.3 as a real, currently published version (alongside
      0.0.1/0.0.2); `gxwf tool-revisions --latest` resolves it to changeset `3ddd99c7efee`. No
      contradiction found; identity and version pin stand as confirmed, not corrected.

      PORT BINDING: the wrapper's real input port is `input_tabular` (gx_data, multiple=true) --
      exactly the shape of flatten_gene_summary_json_to_row's mapped-over `summary_row` output
      (one single-row-plus-header TSV per gene, first column `gene`). Bound
      `identifier_column: 1` (the corpus-confirmed gene-symbol column, now concretely column 1
      per flatten_gene_summary_json_to_row's own column-1-is-`gene` contract) and
      `has_header: 1` (each upstream summary_row TSV carries exactly one header line, per that
      step's own doc). `old_col_in_header` left at the wrapper default (`true`). `fill_char` is
      schema-required with no wrapper default; bound to `""` since this is a uniform single-row
      per-gene join with no expected missing cells across the 10-element collection.
      `include_outputs` left empty (`[]`) -- the optional `output_shell_script` side output is
      not needed. Real output port is `tabular_output` (data, format tabular); `script_output`
      is the include_outputs-gated side output and is not wired anywhere. This discharges the
      `_plan_out` intent: `tabular_output` is the single joined manifest table, matching the
      `lexicmap_9genes_ingestion_manifest.csv` equivalent wired to workflow output
      `ingestion_manifest_all_genes`.
    tool_id: toolshed.g2.bx.psu.edu/repos/iuc/collection_column_join/collection_column_join
    tool_version: "0.0.3"
    tool_shed_repository:
      tool_shed: toolshed.g2.bx.psu.edu
      owner: iuc
      name: collection_column_join
      changeset_revision: 3ddd99c7efee
    in:
      input_tabular: flatten_gene_summary_json_to_row/summary_row
    state:
      input_tabular:
        __class__: ConnectedValue
      identifier_column: 1
      has_header: 1
      old_col_in_header: true
      fill_char: ""
      include_outputs: []
    out:
      - tabular_output

  # ---- N7: extract-by-identifier (Resolution: Identity-pinned) ----
  - id: extract_gene_e_am3_audit
    label: extract_gene_e_am3_audit
    doc: |
      Extract Gene E am3 quarantine audit: element_identifier == "E" from N5's per-gene
      `flagged_tsv` collection (toolshed built-in `__EXTRACT_DATASET__` 1.0.2, resolved via
      `galaxy-tool-cache list`/`summarize` — a bare/stock id, no Tool Shed discovery or
      changeset pin applies).

      PORT CORRECTION vs. prior plan (closes the caveat tracked at open-requirements ledger
      `n7-am3-diagnostic-granularity-unconfirmed`): re-read
      https://github.com/nekrut/disassembler python/lexicmap_streamer.py directly this
      iteration (same commit as before, a51eb58d6c1ca941d3fb8d6adf5e8160c3926b91; repo HEAD
      a3999eb60ae78a804d0200b4a1241038077b6e6a) to settle summary.json vs. flagged.tsv/
      cohort_ledger.tsv for this step's actual need. Evidence:
        - `summary.json`'s only per-haplotype detail is `top_10_haplotypes` (lines 623-697,
          727), built exclusively from `sorted_haplotypes`/`clean_full_haps` — the CLEAN
          cohort. An am3 accession is a premature-stop mutant, so by construction
          (`process_accession_block`, lines 439-480) it is routed to status
          `FLAGGED_PREMATURE_STOPS` / tier 3, never into `clean_full_haps`. summary.json's
          `cohort_breakdown.flagged_premature_stops` (line 713) is an aggregate count only —
          no per-accession record for any flagged accession appears in summary.json at all.
        - `flagged.tsv` (header at line 365; rows written at lines 471/478) carries one row
          per FLAGGED accession with columns `accession, coverage, mean_pident, flag, n_stops,
          stop_codons, hsps_merged` — `stop_codons` (line 430-432) is the actual codon
          position(s) of the internal stop(s) for that accession (e.g. "7" for the paper's
          codon-7 stop), i.e. real per-accession, locus-specific detail, already filtered to
          just the anomalous cohort. `cohort_ledger.tsv` (header line 366, rows line 480)
          carries the same per-accession stop_codons column but for ALL accessions
          (clean+flagged), an unfiltered superset.
        - Conclusion: `flagged_tsv` is the correct port — it is the smallest collection that
          already carries per-accession, locus-specific (codon-position) detail for exactly
          the anomalous/quarantine cohort the am3 audit needs. `cohort_ledger_tsv` would also
          work but requires the consumer to filter by flag, which flagged_tsv already does.
          Neither TSV records the literal nucleotide substitution letters (only the codon
          index), so exact "nt587 G->A" reconstruction still requires the reference sequence
          alongside this table — a residual, non-blocking limitation, not a missing-port
          problem (this step's own job, extracting the Gene E element, is unaffected by it).
        - Rewired this step's `in:`/state from `lexicmap_streamer_tiling_qc/summary_json` to
          `lexicmap_streamer_tiling_qc/flagged_tsv`. This is a same-step port correction (N7
          still reads N5, just a different one of N5's 8 already-declared output ports), not a
          topology repair — no producer node was inserted or removed.
    tool_id: __EXTRACT_DATASET__
    tool_version: "1.0.2"
    in:
      input: lexicmap_streamer_tiling_qc/flagged_tsv
    state:
      input:
        __class__: ConnectedValue
      which:
        which_dataset: by_identifier
        identifier: "E"
    out:
      - output

# Galaxy workflow test file for galaxy-workflow.gxwf.yml (phix174_sra_landscape_spikein_sieve
# / Workflow A). Assembled by implement-galaxy-workflow-test (phase 9) from galaxy-test-plan.yml,
# galaxy-workflow.gxwf.yml, and test-data-refs.json. All job-input and output labels below are the
# literal input/output ids from galaxy-workflow.gxwf.yml's inputs:/outputs: blocks.
#
# Two test cases, matching galaxy-test-plan.yml exactly:
#   1. kmindex_wiring_smoke_generic_fixtures -- wiring/structural smoke test using each pinned
#      Tool Shed wrapper's own real upstream functional-test fixtures (generic, non-phiX174).
#      Existence-only assertions throughout; does not audit the am3 diagnostic.
#   2. gene_e_j_am3_diagnostic_synthetic_lexicmap_index -- the flagship am3 content test, built
#      around a hand-constructed synthetic 3-decoy LexicMap hits fixture (see
#      test-data/synthetic_am3/). All output assertions in this case were verified by directly
#      executing the real vendored lexicmap_streamer.py (galaxy-user-tool.yml, LexicMapStreamer,
#      commit a51eb58d) against that fixture -- the reported values below are actual observed
#      script output, not guessed.
#
# KNOWN BLOCKING GAP (test case 2, carried from galaxy-test-plan.yml unresolved[0]/[1], both
# blocking): `lexicmap_index_selection` in test case 2 is a PLACEHOLDER value
# ("PhiX174E_J_Am3ToyIndex"). No real small kmindex/LexicMap index exists anywhere at test scale
# (109 Logan kmindex shards and up to 25 LexicMap domain indices are multi-terabyte,
# usegalaxy.org-hosted production data only), and no `lexicmap` or `kmindex` CLI was available in
# this authoring environment to build and register one. Consequently:
#   - lexicmap_search (the real Tool Shed tool immediately upstream of lexicmap_streamer_tiling_qc
#     in the concrete workflow) cannot itself be made to emit the synthetic hits table below when
#     `planemo test` actually invokes this workflow end-to-end -- the tests-format `job:` block can
#     only bind values to top-level workflow input labels (confirmed against
#     references/schemas/tests-format.schema.json's `Job`/`TestJob` defs), and
#     `lexicmap_streamer_tiling_qc`'s `lexicmap_results` port is wired to an internal step output
#     (`lexicmap_search/out_file`), not a workflow input -- there is no schema-supported way to
#     inject a pre-built hits table directly onto that port from this file.
#   - The synthetic fixture (test-data/synthetic_am3/{E,J}.synthetic_lexicmap_hits.tsv plus the 3
#     decoy reference FASTAs) is therefore staged as the validated ground truth this test case's
#     assertions were derived from, and as the input a future real-index-build pass (or a direct
#     component/tool-level test of lexicmap_streamer_tiling_qc alone) should reproduce against.
#   - See foundry-feedback.ledger.yml entry
#     `implement-galaxy-workflow-test-no-tests-format-path-to-intermediate-step-input` for the
#     schema-capability gap this exposed, and the plan's own unresolved[0]/[1] for the upstream
#     index-construction gap.
#   - A full `planemo test` run of this test case is therefore expected to fail or hang at
#     `lexicmap_search` today; this is a deliberate, already-documented deferral to phase 11
#     (run-workflow-test), not an authoring omission.

- doc: >-
    kmindex_wiring_smoke_generic_fixtures: structural/wiring smoke test for the kmindex
    containment-screen chain and lexicmap_search's multi-index-selection binding, using each
    pinned Tool Shed wrapper's own real upstream functional-test fixtures (generic, non-phiX174
    sequences and indices). Proves the workflow's collection map-over, multi-select DB/index
    binding, and per-gene sensitivity-override wiring execute without error. Does NOT audit any
    biological content, including the Gene E am3 diagnostic (see the other test case for that).
  job:
    gene_query_panel:
      class: Collection
      collection_type: sample_sheet
      elements:
        - class: File
          identifier: E
          location: "https://raw.githubusercontent.com/galaxyproject/tools-iuc/main/tools/kmindex/test-data/query1.fasta"
          filetype: fasta
        - class: File
          identifier: J
          location: "https://raw.githubusercontent.com/galaxyproject/tools-iuc/main/tools/lexicmap/test-data/lexicmap_query3.fasta"
          filetype: fasta
      # Per-gene LexicMap sensitivity-override columns (align_min_match_pident, etc.) intentionally
      # left unset (no `rows:` block) -- not meaningful for generic non-phiX174 fixture content,
      # per galaxy-test-plan.yml test_cases[0].job_inputs[0].
    # kmindex_db_selection / lexicmap_index_selection removed 2026-09-18: both ports are
    # `multiple: true` selects that reject a delimited string and require an array, which a
    # gxformat2 `text` input cannot carry. The index lists are now literals in the workflow's
    # kmindex_containment_screen / lexicmap_search step state, so they are no longer job inputs.
    kmindex_zvalue: 6
    kmindex_threshold: 0.3
    kmindex_output_format: "json"
    kmindex_fast: false
    lexicmap_top_n_genomes: 0
    lexicmap_advanced_all: true
    tiling_qc_min_coverage: 0.8
    tiling_qc_min_coverage_partial: 0.5
    tiling_qc_min_pident: 60.0
    tiling_qc_max_internal_stops: 0
    tiling_qc_sample_cap: 10
    tiling_qc_allow_frameshifts: false
  outputs:
    kmindex_accession_union:
      # Existence-only: the underlying index has no phiX174 biological relationship, so only
      # "the chain ran and produced non-empty output" is assertable.
      asserts:
        - has_size: { min: 1 }
    clean_full_length_cds_haplotypes:
      class: Collection
      element_tests:
        E:
          asserts:
            - has_size: { min: 1 }
        J:
          asserts:
            - has_size: { min: 1 }
    clean_haplotype_counts:
      class: Collection
      element_tests:
        E:
          asserts:
            - has_n_columns: { min: 2 }
        J:
          asserts:
            - has_n_columns: { min: 2 }
    flagged_audit_reasons:
      class: Collection
      element_tests:
        E:
          # "At least a header line is present."
          asserts:
            - has_n_lines: { min: 1 }
    per_gene_ingestion_summary:
      class: Collection
      element_tests:
        E:
          asserts:
            - has_text: { text: "{" }
        J:
          asserts:
            - has_text: { text: "{" }
    ingestion_manifest_all_genes:
      asserts:
        - has_n_lines: { n: 3, delta: 1 }
    gene_e_am3_quarantine_audit:
      # Existence-only: this case's "E" element is generic fixture sequence, not real phiX174
      # gene E, so no am3 diagnostic content is expected here.
      asserts:
        - has_size: { min: 1 }

- doc: >-
    gene_e_j_am3_diagnostic_synthetic_lexicmap_index: primary functional test for the paper's
    Sanger am3 spike-in diagnostic (freeform-summary.md Stage D2: genome position nt587 G->A
    causes a premature TGG->TAG stop at Gene E codon 7, gpE_W7*). Exercises
    lexicmap_streamer_tiling_qc -> {flatten_gene_summary_json_to_row ->
    join_gene_summary_rows_into_manifest} and -> extract_gene_e_am3_audit against a hand-built
    synthetic 3-decoy LexicMap hits fixture (test-data/synthetic_am3/) seeded with real
    cds/E.fasta and cds/J.fasta content: (1) a Gene-E wildtype-clean decoy (E_wildtype_ctrl,
    sseq == real E.fasta verbatim), (2) a Gene-E am3-positive decoy (E_am3_ctrl, sseq carries
    the exact nt587 G->A / codon-7 TGG->TAG substitution), (3) a Gene-J clean decoy
    (J_clean_ctrl, sseq == real J.fasta verbatim). All assertion values below were confirmed by
    directly executing the real vendored lexicmap_streamer.py against this fixture with the
    workflow's default tiling_qc_* parameters (see test-data/synthetic_am3/*.tsv header
    comments for full provenance). The kmindex chain shares the same job but is not this case's
    focus and is left unasserted (kmindex_accession_union), matching
    galaxy-test-plan.yml omissions[1].

    KNOWN BLOCKING GAP: `lexicmap_index_selection` here is a placeholder
    ("PhiX174E_J_Am3ToyIndex") -- no real small LexicMap index backs it, and the tests-format
    `job:` block cannot bind the synthetic hits fixture directly onto
    lexicmap_streamer_tiling_qc's internal `lexicmap_results` port (see the file header comment
    above and galaxy-test-plan.yml unresolved[0]/[1]). A full `planemo test` run of this case is
    expected to fail/stall at lexicmap_search until a real toy index is constructed and
    registered; that construction is deferred to a later phase.
  job:
    gene_query_panel:
      class: Collection
      collection_type: sample_sheet
      elements:
        - class: File
          identifier: E
          path: test-data/E.fasta
          filetype: fasta
        - class: File
          identifier: J
          path: test-data/J.fasta
          filetype: fasta
      # Per-gene LexicMap sensitivity-override columns, in column_definitions order
      # (align_min_match_pident, align_min_match_len, seed_min_prefix, min_qcov_per_genome).
      # Populated for E (real per-gene values from logan_remaining_runs.json, per
      # test-data-refs.json); left null for J (falls back to wrapper defaults 70/50/17/unset).
      # NOTE: this skill bundle's packaged references do not document a worked `sample_sheet` /
      # `column_definitions` -> tests-format shape; this `rows:` block is a reasoned best-effort
      # construction against the tests-format.schema.json `Collection.rows` field (a dict of
      # column name -> per-row-aligned array) -- see the feedback ledger entry
      # `tests-format-schema-silent-on-sample-sheet-column-definitions-shape`.
      rows:
        align_min_match_pident: [60.0, null]
        align_min_match_len: [35, null]
        seed_min_prefix: [15, null]
        min_qcov_per_genome: [30.0, null]
    # kmindex_db_selection / lexicmap_index_selection removed 2026-09-18: both ports are
    # `multiple: true` selects that reject a delimited string and require an array, which a
    # gxformat2 `text` input cannot carry. The index lists are now literals in the workflow's
    # kmindex_containment_screen / lexicmap_search step state, so they are no longer job inputs.
    kmindex_zvalue: 6
    kmindex_threshold: 0.3
    kmindex_output_format: "json"
    kmindex_fast: false
    lexicmap_top_n_genomes: 0
    lexicmap_advanced_all: true
    tiling_qc_min_coverage: 0.8
    tiling_qc_min_coverage_partial: 0.5
    tiling_qc_min_pident: 60.0
    tiling_qc_max_internal_stops: 0
    tiling_qc_sample_cap: 10
    tiling_qc_allow_frameshifts: false
  outputs:
    clean_full_length_cds_haplotypes:
      # Gene E's clean cohort should contain exactly the wildtype-clean decoy (the am3-positive
      # decoy is routed to FLAGGED, not here). Gene J's clean cohort should contain its one clean
      # decoy. Confirmed via direct script execution: result_E.clean.accessions.fasta contains
      # only E_wildtype_ctrl; result_J.clean.accessions.fasta contains only J_clean_ctrl.
      class: Collection
      element_tests:
        E:
          asserts:
            - has_text: { text: "E_wildtype_ctrl" }
        J:
          asserts:
            - has_text: { text: "J_clean_ctrl" }
    clean_haplotype_counts:
      # Confirmed: header + exactly 1 clean-haplotype row for both E and J.
      class: Collection
      element_tests:
        E:
          asserts:
            - has_n_lines: { n: 2, delta: 1 }
        J:
          asserts:
            - has_n_lines: { n: 2, delta: 1 }
    flagged_accessions:
      # Confirmed: Gene E's flagged FASTA contains exactly the am3-positive decoy.
      class: Collection
      element_tests:
        E:
          asserts:
            - has_text: { text: "E_am3_ctrl" }
    flagged_audit_reasons:
      # Confirmed via direct script execution (result_E.flagged.tsv):
      #   accession        coverage  mean_pident  flag              n_stops  stop_codons  hsps_merged
      #   E_am3_ctrl        1.0000    99.63        PREMATURE_STOPS   1        7            1
      # header + exactly 1 flagged row for E (stop_codons="7", matching the paper's gpE_W7*
      # codon-7 diagnostic). Gene J's flagged.tsv is header-only (0 flagged rows; clean decoy
      # only) -- confirmed exact 1-line file.
      class: Collection
      element_tests:
        E:
          asserts:
            - has_n_lines: { n: 2, delta: 1 }
            - has_text: { text: "7" }
        J:
          asserts:
            - has_n_lines: { n: 1, delta: 0 }
    per_gene_ingestion_summary:
      # Confirmed via direct script execution (result_E.summary.json / result_J.summary.json):
      # gene E total_accessions_evaluated=2, cohort_breakdown.flagged_premature_stops=1;
      # gene J total_accessions_evaluated=1.
      class: Collection
      element_tests:
        E:
          asserts:
            - has_json_property_with_value: { property: "total_accessions_evaluated", value: "2" }
            - has_json_property_with_value: { property: "flagged_premature_stops", value: "1" }
        J:
          asserts:
            - has_json_property_with_value: { property: "total_accessions_evaluated", value: "1" }
    ingestion_manifest_all_genes:
      asserts:
        - has_n_lines: { n: 3, delta: 1 }
        - has_text: { text: "gene" }
    gene_e_am3_quarantine_audit:
      # FLAGSHIP ASSERTION. Gene E element of flagged_tsv (extract_gene_e_am3_audit,
      # by_identifier "E"). Confirmed identical to result_E.flagged.tsv above: exactly 1 flagged
      # row (header + the single am3-positive decoy), identifying E_am3_ctrl with stop_codons="7"
      # -- the workflow-level, assertable analog of the paper's real finding that
      # 2,215,172/2,392,457 (92.59%) of evaluated Gene E accessions carry the nt587 G->A /
      # gpE_W7* substitution (real full-scale numbers recorded only as directional context in
      # test-data-refs.json, not asserted here).
      asserts:
        - has_n_lines: { n: 2, delta: 0 }
        - has_text: { text: "E_am3_ctrl" }
        - has_text: { text: "7" }

status

pass

{
  "command": "gxwf validate galaxy-workflow.gxwf.yml --json --strict",
  "workflow_path": "galaxy-workflow.gxwf.yml",
  "status": "pass",
  "not_run_reason": null,
  "diagnostics": [
    {
      "severity": "major",
      "message": "Fixed during this phase: the workflow's 4 `comments: type: frame` entries (Stage A / Stage B / Stage C / Stage D2) were authored with `label`/`title`/`contains_steps` but no `position`/`size` geometry. Per the phase-9 finding, real Galaxy's `WorkflowCommentModel` requires both fields (HTTP 400 'Field required' on import) even though `gxwf validate` (including `--strict-structure --strict-encoding --strict-state`) reports this workflow clean either way -- confirmed both before and after the fix (`structure_errors: []`, `encoding_errors: []`, identical 5 ok / 0 fail / 4 skip tool-state summary in both runs). gxwf's own schema does not enforce Galaxy's stricter runtime requirement for this field, so a clean `gxwf validate` alone cannot be relied on to catch it.",
      "step_or_field": "comments[0..3] (type: frame)",
      "routes_to_phase": "freeform-summary-to-galaxy-template (phase 5, original author of these frame comments)",
      "status": "fixed_in_place",
      "resolution": "Added `position: [x, y]` and `size: [width, height]` (float-pair arrays, per the packaged galaxy-workflow-comments.md convention) to all 4 frame comments directly in galaxy-workflow.gxwf.yml: Stage A position [0,0] size [400,300]; Stage B position [450,0] size [400,700]; Stage C position [900,0] size [400,500]; Stage D2 position [1350,0] size [400,700] -- non-overlapping, left-to-right by pipeline stage order, each frame's size loosely scaled to its contains_steps count. Values are reasonable placeholder canvas geometry (display-only, no topology/execution effect), not derived from an actual editor layout session."
    },
    {
      "severity": "minor",
      "message": "Tool-state fetch for `toolshed.g2.bx.psu.edu/repos/iuc/kmindex/kmindex_query` (0.6.1+galaxy4) fails to decode (`outputs[1].structure: is missing`) and degrades to `skip_tool_not_found` for step 1 (kmindex_containment_screen). Already tracked as an upstream `@galaxy-tool-util/cli` decoder gap.",
      "step_or_field": "steps[1] kmindex_containment_screen (tool_id toolshed.g2.bx.psu.edu/repos/iuc/kmindex/kmindex_query)",
      "routes_to_phase": "already tracked, not new",
      "status": "known_deferred",
      "resolution": "No action this phase. See feedback ledger entry `tool-util-cli-toolshed-fetch-rejects-real-filtered-list-collection-output` (already open, raised phase 6)."
    },
    {
      "severity": "minor",
      "message": "Tool-state fetch for the 3 authored GalaxyUserTool steps (kmindex_hit_dedup_max_score, lexicmap_streamer, flatten_gene_summary_json_to_row) returns 404 against the Tool Shed (expected: these are local UDTs, never published) and degrades to `skip_tool_not_found` for steps 3, 5, 6.",
      "step_or_field": "steps[3], steps[5], steps[6] (GalaxyUserTool ids)",
      "routes_to_phase": "already tracked, not new",
      "status": "known_deferred",
      "resolution": "Expected behavior for locally-authored UDTs with no tool_shed_repository block. See feedback ledger entry `implement-galaxy-tool-step-udt-binding-undocumented` (already open, raised phase 6) for the undocumented-convention angle."
    },
    {
      "severity": "major",
      "message": "`gxwf validate galaxy-workflow.gxwf.yml --json --connections` (with or without `--strict`) crashes uncaught (`TypeError: step.in is not iterable` in @galaxy-tool-util/cli's toNative.js, exit 1, no JSON emitted at all) on this workflow's ordinary gxformat2 dict-shaped `in:` blocks. Connection/collection-algebra/map-over compatibility checking could not be run for this workflow at all.",
      "step_or_field": "all steps (in: mapping shape, e.g. steps[0].in)",
      "routes_to_phase": "gxwf CLI defect, not a workflow authoring defect",
      "status": "new_diagnostic_reported_upstream",
      "resolution": "Reproduced and filed as feedback ledger entry `gxwf-validate-connections-flag-crashes-uncaught-on-format2-dict-shaped-step-in` (new, this phase). Worked around by validating without `--connections`; collection/map-over shape compatibility is carried forward as a residual runtime risk below."
    }
  ],
  "residual_runtime_risks": [
    {
      "risk": "Collection algebra / map-over compatibility across the 9-step chain (e.g. kmindex's list output into collapse_dataset, lexicmap_search's per-gene mapped output into lexicmap_streamer_tiling_qc's two-input UDT, the sample_sheet gene_query_panel axis threading through multiple steps) was never checked by `gxwf validate --connections` because that flag crashes uncaught on this workflow (see diagnostics). Static validation cannot rule out a shape mismatch here.",
      "proving_artifact": "Phase 11 run-workflow-test (Planemo execution) -- a real invocation will either schedule cleanly or surface `collection_failed`/shape-mismatch invocation messages per the packaged galaxy-workflow-invocation-failure-reference.md."
    },
    {
      "risk": "Tool state for kmindex_containment_screen (real Tool Shed pin, fetch fails to decode) and the 3 GalaxyUserTool steps (no Tool Shed presence at all) was never validated against a real schema by gxwf -- all 4 are `skip_tool_not_found`, not `ok`. The hand-reconstructed parsed_tool for kmindex_query and the UDTs' own tool_id/state binding convention (documented as undocumented in the feedback ledger) remain unverified against a real Galaxy tool-state check.",
      "proving_artifact": "Phase 11 run-workflow-test (Planemo install + invocation) -- tool installation and job scheduling will directly confirm or refute these 4 steps' state shapes."
    },
    {
      "risk": "The frame comments' newly-added `position`/`size` values are reasonable, non-overlapping placeholder canvas geometry (chosen by stage order and step count), not derived from an actual Galaxy workflow-editor layout session. They satisfy the field-presence requirement that caused the phase-9 HTTP 400, but a real Galaxy import has not yet been re-attempted to confirm the comments block as a whole (position/size plus whatever else WorkflowCommentModel requires) now passes real-Galaxy import end to end.",
      "proving_artifact": "A real Planemo-managed Galaxy import (the same `planemo test --test_index 1 --install_galaxy` path phase 9 used) -- this is the only artifact that can confirm real-Galaxy acceptance, since gxwf's own schema does not check this field at all (confirmed clean both before and after the fix)."
    }
  ]
}

• IWC Exemplar Comparison — Workflow A (SRA Landscape & Spike-In Sieve)
• Verdict: No High/Medium-confidence domain exemplar
• Labeled pattern references (Low confidence, NOT domain exemplars)
• 1. `data-fetching/parallel-accession-download` — cleanest map-over→multi-output→flatten skeleton
• 2. `microbiome/mags-building/MAGs-generation` — nearest structural analog for N4/N5/N6
• 3. `microbiome/metagenomic-genes-catalogue` — confirms `pick_value` tool identity
• Structural diff findings, routed by authoring surface
• Open-requirements ledger disposition
• Foundry feedback

# IWC Exemplar Comparison — Workflow A (SRA Landscape & Spike-In Sieve)

Scope: this comparison covers **Workflow A only** (kmindex containment screen → LexicMap
streaming search → LexicMapStreamer multi-HSP tiling/QC/haplotype-calling → per-gene
aggregation → am3 quarantine audit extraction), per `freeform-galaxy-data-flow.md`. Workflows
B/C/D from `freeform-galaxy-interface.md` are out of scope here.

Corpus: `https://github.com/galaxyproject/iwc`, cloned/pulled to `~/.foundry/iwc` on
2026-09-17 (up to date, no new commits since prior clone). Searched with `grep` across all
`.ga` files under `workflows/`, and normalized three candidates with
`gxwf convert --to format2 --compact` for structural inspection.

## Verdict: No High/Medium-confidence domain exemplar

**Confidence: Low (cross-domain structural pattern references only). No nearest domain
exemplar surfaced; `iwc-exemplar.gxwf.yml` is intentionally omitted.**

Walking the skill's feature hierarchy:

1. **Domain/analysis intent** — no match. `grep -rli "kmindex\|lexicmap"` across the entire
   corpus returns zero hits. There is no IWC workflow doing k-mer-index containment screening
   over a large bank of pre-built database shards, nor one doing multi-HSP coordinate tiling /
   codon QC / haplotype collapsing on search hits. The nearest domain neighbors by keyword
   (`kmer|sourmash|mash|containment|sketch`, `sra|accession|metagenom`) are bacterial
   genome-assembly QC, MAG binning, nanopore foodborne-pathogen detection, and generic
   SRA/BioProject fetching — all real metagenomic/microbial workflows, but none screen a fixed
   reference-database bank for containment, and none call haplotypes from multi-HSP tiled hits.
   Domain-blocking per the hierarchy's own rule ("domain comes first ... a structurally similar
   workflow in the wrong science area does not become a misleading exemplar").
2. **Input collection topology** — partial, incidental overlap only (see pattern refs below).
3. **Primary tool families** — no match (kmindex/LexicMap/LexicMapStreamer vs.
   megahit/metaspades/binette/checkm2/sra_tools/fasterq_dump elsewhere in the corpus).
4. **DAG motifs** — the only genuine overlaps are generic Galaxy idioms (map-over-collection,
   flatten, tabular-bridge), not workflow-specific recipes.
5. **Output types/report shape** — no comparable per-gene 8-artifact multi-output emitter.
6. **Test style** — no comparable fixture (see Test-issue routing below).

This is exactly the "fairly unusual IWC domain" case flagged in the run scope — not forced.

## Labeled pattern references (Low confidence, NOT domain exemplars)

Three candidates were converted and inspected. None is presented as *the* nearest exemplar;
each answers one narrow structural question for the template phase.

### 1. `data-fetching/parallel-accession-download` — cleanest map-over→multi-output→flatten skeleton

Full workflow (5 steps) converted via `gxwf convert --to format2 --compact`. Relevant excerpt
(split → per-item tool call with several named outputs → `__APPLY_RULES__` flatten):

```yaml
  - id: fasterq-dump
    tool_id: toolshed.g2.bx.psu.edu/repos/iuc/sra_tools/fasterq_dump/3.1.1+galaxy1
    in:
      - id: input|file_list
        source: Split accessions to collection/list_output_txt
    out:
      - id: list_paired
      - id: log
      - id: output_collection
      - id: output_collection_other
  - id: flatten paired output
    tool_id: __APPLY_RULES__
    in:
      - id: input
        source: fasterq-dump/list_paired
    out:
      - id: output
        add_tags: [{name: PE}]
    tool_state:
      rules:
        mapping:
          - {columns: [1], type: list_identifiers}
          - {columns: [2], type: paired_identifier}
```

Useful as the smallest legible instance of "one tool call emits several named outputs, each
routed through its own reduce" — loosely analogous to N2's map-over-shards and to N5's
multi-artifact-per-element shape, but with only 4 outputs (not 8), no per-element grouping
metadata, and no merge/dedup logic. Labeled as a **pattern reference for N2's map-over
skeleton only**, not a domain match (SRA read download vs. k-mer containment screening).

### 2. `microbiome/mags-building/MAGs-generation` — nearest structural analog for N4/N5/N6

69-step metagenomics MAG assembly/binning/QC workflow. Domain is genome assembly and binning,
not containment screening — cited purely for DAG motifs:

```yaml
  - id: _unlabeled_step_57        # Binette — per-sample bin refinement (map-over)
    tool_id: toolshed.g2.bx.psu.edu/repos/iuc/binette/binette/1.2.1+galaxy0
    out:
      - id: bins
        add_tags: [refined-sample-bins]
  - id: Pool Bins from all samples
    tool_id: __FLATTEN__               # list:list -> list, analogous to N4's index-axis reduce
    in:
      - id: input
        source: _unlabeled_step_57/bins
    out:
      - id: output
        rename: pooled_bins
    tool_state: {join_identifier: "_"}
  ...
  - id: _unlabeled_step_66
    tool_id: toolshed.g2.bx.psu.edu/repos/iuc/collection_column_join/collection_column_join/0.0.3
    in: [{id: input_tabular, source: _unlabeled_step_62}]
    tool_state: {identifier_column: "1", has_header: "1"}
```

Full converted file kept only in the run scratchpad (not published, per Low-confidence rule).
Findings routed below.

### 3. `microbiome/metagenomic-genes-catalogue` — confirms `pick_value` tool identity

Confirms the real Tool Shed identity behind the `conditional-transform-or-pass-through` idiom
`freeform-galaxy-data-flow.md` cites for N4's per-gene sensitivity override:
`toolshed.g2.bx.psu.edu/repos/iuc/pick_value/pick_value/0.2.0` (`first_or_default` style,
`pick_from` list with `default_value`). This was already a named pattern reference in the data
flow brief, not a ledger item, so no ledger action — recorded here as confirmation for the
template phase.

## Structural diff findings, routed by authoring surface

**Template/data-flow issues (for `freeform-summary-to-galaxy-template`):**

- N4's index-axis reduce (`list:list` gene×index → `list` gene) has a real, named built-in
  analog: `__FLATTEN__` (seen pooling MAGs-generation's per-sample bin lists). Template should
  plan to wire this as a concrete built-in step, not a placeholder.
- N3 (kmindex hit-map merge/union) and N6 (JSON-summary → manifest row) both still lack a
  full built-in match, but each is now only a **partial** gap: `Collapse Collection`
  (`nml/collapse_collections`) and `collection_column_join` (`iuc/collection_column_join`)
  are real, named Tool Shed tools that handle the *concatenation*/*tabular-join* half of each
  respectively. The remaining gap narrows to a small custom step (dedup+max-score reduction for
  N3; JSON→TSV-row flattening for N6) feeding into those built-ins, not a from-scratch merge
  tool for the whole node. See new ledger entries below.
- N5's `sample_sheet:record` output-shape recommendation has **no corpus precedent** (zero
  `collection_type: record` hits across all 115 `.ga` files). The corpus's actual idiom for
  "one tool call, many named per-element artifacts" is **N parallel list/sample_sheet
  collections sharing one `element_identifier`**, recombined post hoc via
  `collection_column_join`/`__FLATTEN__` (as MAGs-generation does for bins/contigs/QC). Template
  phase should default to that corpus-precedented shape for N5's 8 outputs unless there's a
  specific reason to prefer the record shape, since the record shape has no IWC test/lint
  precedent to validate against.

**Pattern issues:** none rising to a pattern-page update — the three references above are
recorded inline here, not promoted to a standalone pattern page, since none is a repeatable
domain match (only generic collection-idiom confirmation).

**Tool-step issues:** none new. No corpus wrapper exists for kmindex/LexicMap/LexicMapStreamer;
this is consistent with (not a contradiction of) the existing ledger entries
`kmindex-lexicmap-index-selection-mechanism` and `lexicmapstreamer-wrapper-authoring-pending` —
see note-only updates below.

**Test issues (for later `*-test-to-galaxy-test-plan` phase, not acted on here):** IWC's
accepted-shortcut vocabulary (`iwc-shortcuts-anti-patterns.md`) is directly relevant once test
authoring starts: LexicMap's raw per-gene hit tables are exactly the "stochastic/version-fragile
output" case (like HyPhy's JSON) where existence-only `has_text: "{"`-style probes are accepted;
the deterministic checkpoints this brief already flagged (haplotype counts, ingestion manifest,
am3 quarantine audit JSON) should get real content assertions, per corpus convention. Not
actioned in this phase — flagged for the test-plan Mold.

## Open-requirements ledger disposition

- **Note-updated, status unchanged (still open):** `kmindex-lexicmap-index-selection-mechanism`,
  `lexicmapstreamer-wrapper-authoring-pending` — both got a corpus-search addendum (no
  precedent found); neither is settled by this comparison, both still require inspecting the
  pinned IUC tool schemas / the private `nekrut/disassembler` repo respectively.
- **Newly appended (open, not blocking):** three entries recording structural divergences the
  corpus does not cover — `kmindex-hit-merge-no-corpus-precedent` (N3),
  `n6-json-flatten-no-corpus-precedent` (N6), `n5-output-collection-shape-no-record-precedent`
  (N5). These formalize gaps that previously existed only as prose in
  `freeform-galaxy-data-flow.md` §4/§5, per this skill's mandate to record divergences with no
  corpus precedent as ledger obligations.
- **Untouched:** all other entries (`workflow-scope-boundary-unresolved`,
  `allow-frameshifts-default-unconfirmed`, `host-biome-platform-stratification-script-gap`,
  `nominal-taxonomy-audit-not-modeled-as-step`, `chronaeon-synthetic-metadata-phase0`,
  `chronaeon-hardcoded-statistics-phases`, `wei-2026-dms-dataset-external-sourcing`,
  `hyphaeon-toolshed-status-unverified`, `lexicmapstreamer-implementation-provenance`) — out of
  Workflow A's data-flow scope or not touched by this corpus check.

## Foundry feedback

No feedback ledger entry appended. The skill bundle's procedure, `convert` CLI reference, and
packaged research notes (data-flow contract, shortcuts/anti-patterns, test-data conventions,
open-requirements-ledger protocol) were sufficient to perform the corpus clone, normalize
candidates, rank them, and route findings. Nothing encountered pointed at a defect,
contradiction, or uncovered case in the Foundry skill bundle itself — the difficulty here (no
domain exemplar exists) is a property of this project's unusual tool domain, not the Foundry
asset.

topology_repair:
  escalations: 1
  cap: 5
  open_history: [0]

entries:
  - id: workflow-scope-boundary-unresolved
    status: resolved
    kind: gap
    raised_by: freeform-summary-to-galaxy-interface
    unmet: "Whether the target Galaxy workflow covers only Stages A-D2 (SRA landscape / spike-in sieve, the only content currently in draft_manuscript.tex) or the full Stage A-K project pipeline (through HyphAeon VEP benchmarking, dual-coding shadow-effect calibration, and Fane-mechanics synthesis)"
    missing: "The source freeform-summary explicitly leaves this scoping question open (its Section 7, item 1) rather than resolving it"
    resolved_by: freeform-summary-to-galaxy-data-flow
    supersedes: null
    note: "User-confirmed scope decision (2026-09-17): this pipeline run builds only Workflow A (SRA landscape/spike-in sieve, Stages A/B/C/D2) as a single Galaxy workflow. Workflow B (Stage E, experimental-evolution validation, bowtie2/samtools/mpileup) is a confirmed nice-to-have the user wants built as a separate, later workflow -- out of scope for this run, not abandoned. Workflows C and D (Stages C'/F/G/H and Stage I) are out of scope for this run's data-flow design entirely. This brief modeled the full A-K pipeline shape as labeled stages/candidate subworkflows so nothing was silently dropped before this decision; that record is preserved above for provenance."

  - id: kmindex-lexicmap-index-selection-mechanism
    status: resolved
    kind: gap
    raised_by: freeform-summary-to-galaxy-interface
    step: kmindex_containment_screen
    unmet: "How the 109 kmindex Logan DB shards and the 25 LexicMap domain indices (5-index TARGETED_PHAGE_INDICES subset for per-gene runs) should be exposed as Galaxy tool/workflow-level inputs"
    missing: "The freeform-summary only records the DB/index names as hardcoded Python lists (ALL_KMINDEX_DBS, TARGETED_PHAGE_INDICES); the real kmindex_query/lexicmap_search Tool Shed wrapper's parameter schema has not yet been inspected (summarize-galaxy-tool has not run)"
    resolved_by: "Live verification against usegalaxy.org, 2026-09-18. Both `db_opts|kmindex` (kmindex_query 0.6.1+galaxy4) and `db_opts|lexicmap_index` (lexicmap_search 0.9.0+galaxy1) are `type: select, multiple: true` (tool io_details API). A comma-delimited string is REJECTED at parameter validation -- POSTing db_opts|kmindex=\"GENOMIC_PHG,GENOMIC_VRL\" returns \"Parameter 'kmindex': an invalid option ('GENOMIC_PHG,GENOMIC_VRL') was selected\", because a multi-select reads the whole string as ONE option value. The JSON array [\"GENOMIC_PHG\",\"GENOMIC_VRL\"] is accepted and fans out correctly: job bbd44e69cb8906b5eadd64618b449770 in history bbd44e69cb8906b57012ea23efccb56e completed with 32 GENOMIC_PHG + 33 GENOMIC_VRL result datasets. Since a gxformat2 `text` workflow input cannot carry an array, the delimited-string modeling this entry assumed is unusable: the `kmindex_db_selection` and `lexicmap_index_selection` text inputs were REMOVED and the 109 shard names / 5 TARGETED_PHAGE_INDICES are now literal lists in the kmindex_containment_screen and lexicmap_search step state. Trade-off accepted: index sets are no longer settable at invocation time and require editing the workflow."
    supersedes: null
    note: "This brief models index selection as a controlled-vocabulary select/multi-select parameter carrying the known DB/index name lists. Must be reconciled against the actual tool XML during implement-galaxy-tool-step / discover-shed-tool. [freeform-summary-to-galaxy-data-flow, Workflow A only] Cross-checked against the newly confirmed LexicMapStreamer provenance (see lexicmapstreamer-implementation-provenance): the nekrut/disassembler repo implements the multi-HSP tiling/QC client, not the kmindex_query/lexicmap_search Tool Shed wrappers' own DB/index parameter schema, so that discovery does not bear on this entry. Still open pending inspection of the pinned IUC changesets themselves (kmindex_query 0.6.1+galaxy4, lexicmap_search 0.9.0+galaxy1). [compare-against-iwc-exemplar, 2026-09-17] Corpus check (galaxyproject/iwc, 115 workflows) found no workflow exposing a >20-way controlled-vocabulary shard/index select as a workflow input or map-over axis -- the largest comparable case, data-fetching/parallel-accession-download's split_file_to_collection, dynamically chunks an arbitrary-length input file rather than enumerating a fixed named list, a different mechanism. No corpus precedent to draw on; still open, unchanged. [freeform-summary-to-galaxy-template, 2026-09-17] Topology settled `kmindex_db_selection` (N2) and `lexicmap_index_selection` (N4) as plain multi-value `text` workflow inputs consumed entirely inside each tool's own call -- kmindex_containment_screen's declared output is a `list` collection (one element per selected shard) and lexicmap_search's declared output is a `list:list` collection (gene outer axis from the real sample_sheet map-over, index inner axis from the tool's own per-index output declaration), rather than modeling either selection as a second Galaxy-level map-over collection. This is a topology commitment (see galaxy-workflow-draft.gxwf.yml steps kmindex_containment_screen / lexicmap_search); the exact tool-side input widget realizing it is still open and unchanged by this note. [advance-galaxy-draft-step, 2026-09-17] REFUTED in part: summarize-galaxy-tool on the pinned changeset (iuc/lexicmap/lexicmap_search 0.9.0+galaxy1, bcb6caec41eb) shows the wrapper's `parsed_tool` declares exactly one output, `out_file` (plain `data`, format `tabular`) -- no `discover_datasets`, no per-index collection output. There is no 'tool's own per-index output declaration'; that specific claim above is struck. `lexicmap_index` (case `db_opts_selector: db`, 'Locally installed LexicMap indexes') is a `gx_select` with `multiple: true` and dynamic `options: null` -- a single job accepts the full multi-value index selection and returns one aggregate hits table, so lexicmap_search's real per-gene output shape (mapped over `gene_query_panel`) is a flat `list`, not `list:list`. The kmindex/lexicmap side of the index-selection-mechanism question (delimited-text -> multi-select binding; dynamic option population) remains open and unchanged. The invalidated inner-axis/list:list claim's downstream consequence (flatten_lexicmap_results_by_gene's `__FLATTEN__` step assumed a nested input) is tracked separately as blocking entry `lexicmap-search-flatten-shape-mismatch`, superseded by this note. [advance-galaxy-draft-step, 2026-09-17] kmindex side CONFIRMED by direct XML inspection: galaxy-tool-cache add/summarize cannot parse the pinned iuc/kmindex/kmindex_query changeset (0.6.1+galaxy4, b6fa25b6b436) or any other galaxy-suffixed version -- a reproducible cache/parser defect (see galaxy-tool-summary.json warnings[] and the feedback ledger) -- so the wrapper's real upstream XML was read directly (tools-iuc commit 7681be7f40, confirmed via the Tool Shed API's remote_repository_url for this exact changeset) in place of the normal automated summarize-galaxy-tool path. Result: `db_opts|kmindex` (case `db_opts_selector: db`) is a `gx_select` with `multiple: true`, dynamic `options: null` (populated from the `kmindex` Tool Shed data table) -- the same shape as lexicmap_index, confirming this is a native Galaxy multi-select in both wrappers, not a repeat or per-element data input. The command template loops over indices internally (`#for $i, $INDEX in enumerate($INDICES)` on `$db_opts.kmindex.fields.path.split(',')`) and the `output` collection output (`list`, discover_datasets over `query_output/*.json`) fans out to one JSON element per selected shard -- matching the already-settled topology commitment exactly, no scatter/map-over needed. kmindex_containment_screen implemented on this basis this iteration. Still open and non-blocking: whether the scalar `kmindex_db_selection` `text` workflow input's delimited-string value is accepted as-is by a `multiple: true` select at runtime, or needs adaptation to a literal array -- this is a runtime parameter-shape question, not a topology gap, and does not block draft-validate."

  - id: allow-frameshifts-default-unconfirmed
    status: resolved
    kind: gap
    raised_by: freeform-summary-to-galaxy-interface
    step: multi_hsp_tiling_qc
    unmet: "Production default of stream_lexicmap_msa.py's --allow-frameshifts flag"
    missing: "Source prose states the default/production setting could not be confirmed from prose alone"
    resolved_by: advance-galaxy-draft-step
    supersedes: null
    note: "Exposed in this brief as an explicit boolean workflow parameter with no default asserted, rather than assuming reject-by-default. [freeform-summary-to-galaxy-data-flow] User-confirmed (2026-09-17): stream_lexicmap_msa.py's multi-HSP tiling/QC logic (\"LexicMapStreamer\") is not custom glue authored for this project from scratch -- it is backed by a real, existing implementation at https://github.com/nekrut/disassembler (private repo). The --allow-frameshifts production default is therefore expected to be readable directly from that repo's source, not an unrecoverable prose ambiguity. Still open (the value itself has not been read/confirmed) -- tracked as concrete follow-up work in the new lexicmapstreamer-wrapper-authoring-pending entry, to be resolved when that repo is inspected during wrapper authoring. [freeform-summary-to-galaxy-template, 2026-09-17] Carried into galaxy-workflow-draft.gxwf.yml as workflow input `tiling_qc_allow_frameshifts`: type boolean, `optional: true`, no `default:` -- a legal, non-TODO topology expression of \"value genuinely unknown,\" distinct from a topology TODO. Still open. [advance-galaxy-draft-step, 2026-09-17] RESOLVED: the private repo https://github.com/nekrut/disassembler was cloned and python/lexicmap_streamer.py read directly. Its argparse declares `parser.add_argument(\"--allow-frameshifts\", action=\"store_true\", help=...)` with no `default=True` -- Python's argparse gives a bare `action=\"store_true\"` flag an implicit default of `False`. Confirmed production default: OFF/False. The workflow input `tiling_qc_allow_frameshifts` remains `optional: true` with no asserted default at the workflow-input level (a user can still override per-run), but the wrapper itself (galaxy-user-tool.yml, tool_id `lexicmap_streamer`) now reflects the real tool's off-by-default semantics."

  - id: host-biome-platform-stratification-script-gap
    status: open
    kind: dropped
    raised_by: freeform-summary-to-galaxy-interface
    units: "Stage D host/biome/platform metadata stratification of the 2,114,904-accession Logan hit set (the paper's Fig 1B/C percentages)"
    because: "freeform-summary Section 5 (Stage D) states no corresponding script was identified among the root-level .py files; likely produced via ad hoc NCBI Entrez/SRA-metadata queries or manual curation not captured in the repo snapshot"
    unmet: "A reproducible Galaxy step that recomputes the biome/platform stratification numbers"
    missing: "No discoverable tool or script to wrap for this stratification"
    resolved_by: null
    supersedes: null
    note: "Not modeled as an executable workflow step in this brief; recorded as a labelled gap rather than invented as a fabricated step."

  - id: nominal-taxonomy-audit-not-modeled-as-step
    status: resolved
    kind: dropped
    raised_by: freeform-summary-to-galaxy-interface
    units: "Stage D nominal-taxonomy SRA metadata audit (136 runs / 18 BioProjects, manual classification into Experimental Evolution / Platform Benchmarking / Paleogenomics Control / etc.), including the candidate `Nominal-taxonomy BioProject classification table` declared workflow input"
    because: "Source describes this as a manual/registry-based certification exercise, not a computational tool invocation over data the workflow would produce"
    unmet: "N/A — not intended as an executable Galaxy step"
    missing: "No tool; this is curated reference/provenance data, not a computation"
    resolved_by: freeform-summary-to-galaxy-template
    supersedes: null
    note: "Modeled in this brief as a static reference-data input (a BioProject classification table) rather than a workflow step. Downstream data-flow phase should confirm this framing. [freeform-summary-to-galaxy-data-flow, 2026-09-17] Confirmed: no Workflow A node (N1-N7) consumes this table; carried forward as an open framing question (data-flow brief section 7, item 2) rather than dropped outright. [freeform-summary-to-galaxy-template, 2026-09-17] Resolved: the settled N1-N7 topology has no node wired to this table, and per the data-flow brief's own suggestion (\"an unconnected input is unusual for a runnable workflow ... consider dropping it\"), this Mold drops it from galaxy-workflow-draft.gxwf.yml's declared workflow inputs entirely. It remains documentation/provenance context (see this workflow draft's top-level `doc:`), not a Galaxy input or step."

  - id: chronaeon-synthetic-metadata-phase0
    status: open
    kind: dropped
    raised_by: freeform-summary-to-galaxy-interface
    units: "run_chronaeon_phix174_sieve.py Phase 0 fabricated pseudo-collection_date construction feeding hyphaeon autoclock"
    because: "freeform-summary Section 7, item 7 confirms by direct script inspection that Phase 0's collection_date values are hash-derived synthetic placeholders, not real SRA/BioSample metadata"
    unmet: "A genuine collection-date-bearing input to the autoclock step"
    missing: "No real per-accession collection_date source was identified in this repo snapshot"
    resolved_by: null
    supersedes: null
    note: "This brief models autoclock's date-bearing metadata input as an open/external requirement (real SRA BioSample collection dates to be sourced) rather than porting the synthetic generator as if it were a genuine step."

  - id: chronaeon-hardcoded-statistics-phases
    status: open
    kind: dropped
    raised_by: freeform-summary-to-galaxy-interface
    units: "run_chronaeon_phix174_sieve.py Phases 2, 6, 7, 8 (e.g. the Gene D/E and Gene B/A Mann-Whitney medians/p-values, and a per-gene VEP correlation table)"
    because: "freeform-summary Section 7, item 7 confirms these values appear as hardcoded constants written directly into the script, not recomputed from input data at run time"
    unmet: "Genuine, re-executable recomputation of the statistics these phases report"
    missing: "The phases as written are reporting/munging of pre-baked numbers, not real computations"
    resolved_by: null
    supersedes: null
    note: "Only Phases 1/3/4 (the confirmed genuine hyphaeon autoclock/r0/meme CLI invocations) are modeled as real workflow steps in this brief. Phases 2/6/7/8 are excluded pending reimplementation as genuine pandas/scipy steps over live inputs."

  - id: wei-2026-dms-dataset-external-sourcing
    status: open
    kind: gap
    raised_by: freeform-summary-to-galaxy-interface
    unmet: "Concrete, fetchable source location for the Wei, Li & Lehner (2026) whole-genome DMS ground-truth table, a hard input dependency for Stages E/G/H/J"
    missing: "Dataset is not present in the repo; every downstream script expects it at a fixed local path (phix174_WGM/data/...zip) that does not exist in this checkout"
    resolved_by: null
    supersedes: null
    note: "Modeled in this brief as an external reference-data input (bioRxiv doi:10.64898/2026.07.25.740675 supplement / GitHub, exact download mechanism TBD). Must be resolved before test-data resolution phases can fixture it."

  - id: hyphaeon-toolshed-status-unverified
    status: open
    kind: gap
    raised_by: freeform-summary-to-galaxy-interface
    unmet: "Whether HyphAeon (github.com/veg/hyphaeon, CLI + Python package) and the companion hyphaeon_overlaps package are already Tool-Shed-wrapped, conda/pip-installable, or need to be authored from scratch as Galaxy tools"
    missing: "freeform-summary Section 7, item 3 flags this as unverified from the repo alone"
    resolved_by: null
    supersedes: null
    note: "This brief assumes new Galaxy tool wrappers will be authored for both packages; discover-shed-tool should check Tool Shed availability before author-galaxy-tool-wrapper is invoked for either."

  - id: lexicmapstreamer-implementation-provenance
    status: resolved
    kind: gap
    raised_by: freeform-summary-to-galaxy-data-flow
    step: lexicmap_streamer_tiling_qc
    unmet: "Whether stream_lexicmap_msa.py's multi-HSP coordinate tiling / codon QC / haplotype-collapsing logic (the user's \"LexicMapStreamer\") has any existing implementation to wrap, or must be authored as new Galaxy tool code from scratch"
    missing: "freeform-summary's tool inventory (Section 5, 'Custom/internal orchestration code') characterized stream_lexicmap_msa.py as code with no separate installable tool"
    resolved_by: freeform-summary-to-galaxy-data-flow
    supersedes: null
    note: "User-confirmed (2026-09-17): LexicMapStreamer already exists as a real, working implementation in the private repository https://github.com/nekrut/disassembler -- this is the actual code behind stream_lexicmap_msa.py's multi-HSP tiling/QC logic, not something to author from scratch by reverse-engineering the summary's prose. This does not itself discharge the wrapper-authoring obligation -- see lexicmapstreamer-wrapper-authoring-pending for the remaining open work."

  - id: lexicmapstreamer-wrapper-authoring-pending
    status: resolved
    kind: gap
    raised_by: freeform-summary-to-galaxy-data-flow
    step: lexicmap_streamer_tiling_qc
    unmet: "A Galaxy tool wrapper (XML + macros, or a Planemo-testable CLI wrapper) for LexicMapStreamer's multi-HSP tiling/QC/haplotype-collapsing step (Workflow A, node N5)"
    missing: "No Tool Shed wrapper is known to exist yet; the real implementation lives in the private repo https://github.com/nekrut/disassembler and its CLI/API surface, install method, and license have not been inspected in this run"
    resolved_by: advance-galaxy-draft-step
    supersedes: null
    note: "Not a from-scratch authoring problem (see lexicmapstreamer-implementation-provenance) -- discover-shed-tool should first check whether a wrapper already exists for this repo; if not, author-galaxy-tool-wrapper should target the disassembler repo's actual CLI/API rather than reverse-engineering behavior from stream_lexicmap_msa.py prose alone. The --allow-frameshifts default (see allow-frameshifts-default-unconfirmed) and the exact shape of N5's 8 per-gene output artifacts (see freeform-galaxy-data-flow.md §2, node N5) should both be confirmed by reading this repo's source during that authoring pass. [compare-against-iwc-exemplar, 2026-09-17] Corpus search for kmindex/lexicmap/comparable multi-HSP-tiling or haplotype-calling tools returned zero hits across the full IWC corpus -- confirms no shortcut via an existing IWC-adjacent wrapper; the nekrut/disassembler-backed wrapper-authoring path remains the only route. Still open, unchanged. [freeform-summary-to-galaxy-template, 2026-09-17] Topology settled: `lexicmap_streamer_tiling_qc` in galaxy-workflow-draft.gxwf.yml declares all 8 output ports concretely (see n5-output-collection-shape-no-record-precedent, resolved) and is mapped one call per gene. Still open on wrapper identity/authoring; also now the anchor for the am3-diagnostic-granularity question raised in n7-am3-diagnostic-granularity-unconfirmed, since that question turns on the same repo's summary.json / flagged.tsv schema. [advance-galaxy-draft-step, 2026-09-17] RESOLVED: discover-shed-tool re-confirmed the miss this iteration (3 query variants, 0 hits; galaxy-tool-pin.json). https://github.com/nekrut/disassembler was cloned (git clone succeeded; the repo, while described by the user as private, was accessible with this environment's ambient `gh`/git credentials -- no access blocker encountered). python/lexicmap_streamer.py (817 lines) was read directly: pure Python 3 stdlib, no third-party imports; full argparse CLI confirmed (see galaxy-user-tool.yml for the complete flag table). author-galaxy-tool-wrapper produced galaxy-user-tool.yml: `GalaxyUserTool` id `lexicmap_streamer`, version `1.0.0`, container `python:3.13-slim`, the script vendored verbatim (MIT license, (c) Anton Nekrutenko, commit a51eb58) as a `configfiles` entry. All 8 of N5's declared output ports map 1:1 onto the script's real fixed-suffix output files (`.clean.msa.fasta`, `.clean.haplotypes.tsv`, `.clean.accessions.fasta`, `.clean_expanded.accessions.fasta`, `.flagged.accessions.fasta`, `.flagged.tsv`, `.cohort_ledger.tsv`, `.summary.json`) -- no invented outputs, no port left unresolved; `--no-a2m` passed to suppress the one script output (`.a2m`) with no corresponding declared port. galaxy-workflow-draft.gxwf.yml's `lexicmap_streamer_tiling_qc` step now carries this concrete `tool_id`/`tool_version`/`in`/`state`/`out`. CORRECTION surfaced by this reading: the step's `_plan_in` (single semantic port) undercounted the real input surface -- the script requires the per-gene reference CDS FASTA (`-q/--ref`) in addition to the LexicMap hit table; wired to `gene_query_panel`, the same collection `lexicmap_search` already consumes, so no new workflow input or topology change was needed, just an added `in:` entry. `gxwf draft-validate --concrete` returns `draft valid` / `Concrete: OK` on the mutated draft (this step's tool_id resolves to a `skip_tool_not_found` Tool Shed-lookup skip, exactly like the pre-existing kmindex_query skip, not a hard failure)."

  - id: reference-genome-workflow-input-dropped
    status: resolved
    kind: dropped
    raised_by: freeform-summary-to-galaxy-template
    units: "`Reference genome (NC_001422.1)` as a declared Workflow A workflow-level input"
    because: "The settled N1-N7 topology (galaxy-workflow-draft.gxwf.yml) has no node that consumes the whole-genome FASTA -- only the per-gene CDS FASTAs in `gene_query_panel` are used by N1 (combine) and N4 (LexicMap search), matching freeform-summary.md's own language about the 'canonical ΦX174 gene coordinate frame.' freeform-galaxy-data-flow.md §1 and §7 item 1 already flagged this input as unwired and recommended dropping it from Workflow A's declared inputs if the template phase found no consumer, which is what this Mold confirmed."
    unmet: "N/A -- not a computational dependency of any Workflow A step"
    missing: "No node in Workflow A reads the whole-genome FASTA"
    resolved_by: freeform-summary-to-galaxy-template
    supersedes: null
    note: "Dropped from galaxy-workflow-draft.gxwf.yml's declared `inputs:`; recorded in the draft's top-level `doc:` for provenance. This input may belong to Workflow B (Stage E), where `NC_001422.1:2395-2919` is genuinely used for bowtie2/samtools alignment -- out of scope for this run."

  - id: n7-gene-e-audit-source-resolved
    status: resolved
    kind: gap
    raised_by: freeform-summary-to-galaxy-data-flow
    step: extract_gene_e_am3_audit
    unmet: "Whether N7's Gene E am3 quarantine audit is literally the Gene E element of N5's (lexicmap_streamer_tiling_qc's) per-gene summary.json sample_sheet, or a ninth, separate per-gene artifact N5 also emits"
    missing: "freeform-galaxy-data-flow.md §2 N5/N7 and §7 item 3 left this open, resolvable only once the nekrut/disassembler-backed wrapper's real output surface is known"
    resolved_by: freeform-summary-to-galaxy-template
    supersedes: null
    note: "TOPOLOGY DECISION (this Mold must never leave a topology choice as TODO): modeled N7 as an EXTRACTION of the Gene E element of N5's summary.json collection (collection-unbox-singleton / __EXTRACT_DATASET__ by element_identifier == \"E\"), not a distinct ninth artifact. Rationale: freeform-summary.md Stage D2 states the am3 detection logic 'lives inside the QC step of stream_lexicmap_msa.py' (i.e. inside N5) and is only 'summarized in results/02_layer1_quasispecies/gene_e_am3_quarantine_audit.json' -- a path under ChronAeon Phase 2 (freeform-summary.md Stage I), a stage this run's scope decision (workflow-scope-boundary-unresolved) places out of scope for Workflow A, and which freeform-summary.md §7 item 7 / chronaeon-synthetic-metadata-phase0 / chronaeon-hardcoded-statistics-phases flag as partly fabricated-metadata / hardcoded-statistics rather than a reliable recomputation path. Treating N7 as a literal extraction of N5's own Gene E output avoids manufacturing a dependency on that unreliable, out-of-scope stage, and matches the source's own statement that the detection is internal to N5's QC rather than a separate computation. See galaxy-workflow-draft.gxwf.yml step `extract_gene_e_am3_audit`'s `_plan_state` for the full reasoning as carried in the draft itself. This resolution carries one caveat, tracked separately: n7-am3-diagnostic-granularity-unconfirmed."

  - id: n7-am3-diagnostic-granularity-unconfirmed
    status: resolved
    kind: gap
    blocking: false
    raised_by: freeform-summary-to-galaxy-template
    step: extract_gene_e_am3_audit
    unmet: "Whether N5's (lexicmap_streamer_tiling_qc's) per-gene summary.json / flagged.tsv actually carries locus-specific detail (genome position nt587, G->A, codon 7 TGG->TAG) sufficient to reconstruct the paper's '2,215,172/2,392,457 (92.59%) of evaluated Gene E accessions carry this substitution' statistic, as opposed to only an aggregate premature-internal-stop count"
    missing: "freeform-summary.md describes stream_lexicmap_msa.py's QC step as flagging premature internal stop codons generally (via codon-table translation) and describes summary.json as carrying 'quantile distributions, regime diagnostics, multi-HSP recovery counts' -- it does not explicitly confirm the summary.json (or flagged.tsv) schema records the specific substituted nucleotide/codon per flagged accession, only that a stop was found"
    resolved_by: advance-galaxy-draft-step
    supersedes: null
    note: "Not treated as blocking: the underlying computation (per-accession codon-level translation and stop-codon detection) is plausibly already granular enough, since stream_lexicmap_msa.py must inspect individual codons to flag a stop at all -- the open question is only whether that per-accession locus detail is surfaced into N5's declared outputs (summary.json / flagged.tsv) or needs an additional field. To be confirmed during the same nekrut/disassembler wrapper-authoring pass tracked by lexicmapstreamer-wrapper-authoring-pending; if the real repo's outputs do not carry this detail, author-galaxy-tool-wrapper / implement-galaxy-tool-step should add it as a wrapper-level enhancement rather than escalate a topology repair, since the wiring (N7 reads N5's per-gene summary output) remains correct either way. [advance-galaxy-draft-step, 2026-09-17] PARTIALLY CONFIRMED, still open, still non-blocking, but the concern sharpens rather than dissolves: python/lexicmap_streamer.py was read directly. `summary.json` (`summary_data` dict) carries only aggregate quantiles/cohort counts/Shannon entropy plus a `top_10_haplotypes` preview limited to the CLEAN FULL-LENGTH cohort (ranked by count, with `aa_mutations`/`nt_mutations` string fields like `W7*` / `587G>A` that DO encode exact codon/nt position when present) -- it carries no per-accession stop-codon locus list. The literal per-accession, 1-based codon-position detail (`stop_codons` column, e.g. \"2,15\") lives in `flagged.tsv` and `cohort_ledger.tsv`, which N5 also emits as separate declared output ports (`flagged_tsv`, `cohort_ledger_tsv`) but which N7 (`extract_gene_e_am3_audit`, per n7-gene-e-audit-source-resolved) does NOT read -- N7 is wired only to `summary_json`. Since am3 is specifically a PREMATURE STOP mutant, an am3-carrying accession is by construction routed to the FLAGGED cohort (not the CLEAN top_10_haplotypes preview), so summary.json's per-haplotype mutation strings will not surface it either. Net: if the Gene E am3 quarantine audit needs the exact per-accession locus (not just the aggregate `flagged_premature_stops` count that summary.json does carry), N7 is reading the wrong port -- it should read `flagged_tsv` (or `cohort_ledger_tsv`) instead of/in addition to `summary_json`. Left open and non-blocking per this entry's own standing guidance (a topology question for the template tier, out of scope for this single-step iteration, which was scoped to lexicmap_streamer_tiling_qc only) rather than escalated to repair-galaxy-draft-topology on this pass. [advance-galaxy-draft-step, 2026-09-17] RESOLVED while implementing extract_gene_e_am3_audit itself: re-read python/lexicmap_streamer.py (same commit a51eb58d6c1ca941d3fb8d6adf5e8160c3926b91) and re-confirmed the above analysis line-by-line (process_accession_block lines 400-480; flagged.tsv header/rows at lines 365/471/478; cohort_ledger.tsv header/rows at lines 366/480; summary.json construction at lines 599-730). Rewired extract_gene_e_am3_audit's `in:` from `lexicmap_streamer_tiling_qc/summary_json` to `lexicmap_streamer_tiling_qc/flagged_tsv` -- flagged_tsv is the smallest N5 output already filtered to the anomalous/quarantine cohort and already carrying a per-accession `stop_codons` (codon-position) column, which is exactly the per-accession, locus-specific detail this audit needs, whereas cohort_ledger_tsv is an unfiltered superset requiring the consumer to filter by flag itself. This was a same-step port correction (N7 still reads N5, just the correct one of its 8 already-declared output ports), not a topology repair -- no producer node was inserted or removed, so repair-galaxy-draft-topology was not invoked. One residual, explicitly non-blocking limitation remains: neither flagged.tsv nor cohort_ledger.tsv records the literal nucleotide-substitution letters (only the codon index), so reconstructing the paper's exact 'nt587 G->A' phrasing still requires cross-referencing the reference sequence -- this does not affect the correctness of this step's own job (extracting the Gene E element), only a downstream reporting nicety."

  - id: kmindex-hit-merge-no-corpus-precedent
    status: resolved
    kind: gap
    blocking: false
    raised_by: compare-against-iwc-exemplar
    step: kmindex_hit_union
    unmet: "A concrete Galaxy tool/idiom for N3's per-shard kmindex hit-map merge: dedup accessions across ~109 shard JSON files while keeping the max containment score per accession"
    missing: "IWC corpus (galaxyproject/iwc, cloned/pulled 2026-09-17, 115 workflows) has no workflow performing key-based dedup-with-max-score reduction over a list collection. The closest built-in idiom found, Collapse Collection (toolshed.g2.bx.psu.edu/repos/nml/collapse_collections/collapse_dataset, used in microbiome/mags-building/MAGs-generation.ga steps 21-22 and microbiome/metagenomic-genes-catalogue.ga), performs plain concatenation of a list collection into one dataset with no per-key reduction -- it could serve as a first pass (concatenate, then a custom sort/dedup script) but does not implement the max-score-keeping semantics itself."
    resolved_by: freeform-summary-to-galaxy-template
    supersedes: null
    note: "Was carried only as prose in freeform-galaxy-data-flow.md section 4 item 1, never previously promoted to this ledger. Promoted here per compare-against-iwc-exemplar's mandate to record structural divergences with no corpus precedent. Template phase should plan a small custom aggregation step (e.g. a Python/pandas tool) fed by Collapse Collection's concatenated output, rather than expect a single built-in tool to do both. [freeform-summary-to-galaxy-template, 2026-09-17] Resolved: galaxy-workflow-draft.gxwf.yml wires this as two concrete steps -- `kmindex_hit_concat` (Identity-pinned, nml/collapse_collections/collapse_dataset) followed by `kmindex_hit_dedup_max_score` (Deferred, tool_id TODO, _plan_state documents the dedup+max-score reduction). No blocking gap remains; the custom step's wrapper identity is left open for discover-shed-tool / author-galaxy-tool-wrapper, which is an ordinary Deferred-tier obligation, not an unmet topology need. [advance-galaxy-draft-step, 2026-09-17] Wrapper identity now settled: discover-shed-tool re-confirmed the miss (datamash groupby/max rejected -- tabular/pre-sorted input only, cannot parse JSON); author-galaxy-tool-wrapper produced galaxy-user-tool-kmindex-hit-dedup-max-score.yml (GalaxyUserTool, id kmindex_hit_dedup_max_score, version 1.0.0). kmindex_hit_dedup_max_score is now fully concrete in the draft (tool_id/tool_version mirror the UDT identity, ports concatenated_hits/accession_union). No blocking entry raised -- the step's single wired input (kmindex_hit_concat/output) is sufficient to compute its declared output."

  - id: n6-json-flatten-no-corpus-precedent
    status: resolved
    kind: gap
    blocking: false
    raised_by: compare-against-iwc-exemplar
    step: ingestion_manifest_aggregation
    unmet: "A Galaxy tool/idiom to flatten each gene's nested .summary.json (N5 output) into one manifest row before concatenation into Ingestion manifest (all genes)"
    missing: "IWC corpus has a real, directly relevant built-in tool for the tabular half of this bridge -- collection_column_join (toolshed.g2.bx.psu.edu/repos/iuc/collection_column_join/collection_column_join/0.0.3, used in MAGs-generation.ga steps 66-67 and metagenomic-genes-catalogue.ga) joins a list collection of per-element tabular files by an identifier column into one combined table. No corpus workflow performs the JSON-to-tabular-row flattening half; every per-sample tabular input to collection_column_join in the corpus is already tabular going in (e.g. CoverM/Quast/CheckM2 reports), never raw JSON."
    resolved_by: freeform-summary-to-galaxy-template
    supersedes: null
    note: "Confirms freeform-galaxy-data-flow.md section 4 item 5's claim ('no generic Galaxy tool flattens an arbitrary nested per-gene JSON into one manifest row') is correct for the flatten step, but narrows the remaining gap: only a small per-gene JSON-to-TSV-row flattening tool (jq/python) needs to be authored; the subsequent collection-to-table join can reuse the real, named collection_column_join Tool Shed tool rather than a bespoke merge. [freeform-summary-to-galaxy-template, 2026-09-17] Resolved: galaxy-workflow-draft.gxwf.yml wires this as two concrete steps -- `flatten_gene_summary_json_to_row` (Deferred, tool_id TODO, mapped over the gene collection) followed by `join_gene_summary_rows_into_manifest` (Identity-pinned, iuc/collection_column_join/collection_column_join/0.0.3). No blocking gap remains. [advance-galaxy-draft-step, 2026-09-17] Deferred obligation discharged: `gxwf tool-search \"json to tabular\"` found only generic JSON tools (best hit `iuc/jq`, confirmed installable via galaxy-tool-cache add/summarize, changeset 4e62e523c2b6) -- rejected because its `filter`/`arguments` parameters cannot read the mapped input's Galaxy `element_identifier` without an extra `collection_element_identifiers`-style producer step and port, which would be an avoidable topology change. Authored a small GalaxyUserTool instead (`galaxy-user-tool-flatten-gene-summary.yml`, id `flatten_gene_summary_json_to_row`, version `1.0.0`, pure Python 3 stdlib, container `python:3.13-slim`) whose `shell_command` reads `$(inputs.summary_json.element_identifier)` directly -- confirmed valid against the installed `@galaxy-tool-util/schema` package's `gx-data.js` (`job_runtime` File-object shape declares `element_identifier: S.optional(S.String)` alongside `path`; also flagged as an authoring-note documentation gap in the feedback ledger, entry `galaxy-user-tool-authoring-missing-element-identifier-expression`). Output is a two-line TSV (header + one data row), first column `gene` = element_identifier, remaining columns the flat scalar fields of summary.json (regime diagnostics, cohort-breakdown counts, multi-HSP recovery counts, coverage/pident quantiles), read directly from this run's own vendored lexicmap_streamer.py rather than reverse-engineered from prose. `gxwf validate-tool-source galaxy-user-tool-flatten-gene-summary.yml` returns `OK`; `gxwf draft-validate --concrete galaxy-workflow-draft.gxwf.yml` returns `draft valid` / `Concrete: OK` on the mutated draft. join_gene_summary_rows_into_manifest's stale `in:` reference to this step's placeholder port was updated to the real `summary_row` port name (same-step reference fix, not a topology repair; see that step's own `_plan_state`)."

  - id: n5-output-collection-shape-no-record-precedent
    status: resolved
    kind: gap
    blocking: false
    raised_by: compare-against-iwc-exemplar
    step: lexicmap_streamer_tiling_qc
    unmet: "Whether LexicMapStreamer's 8 per-gene output artifacts should be modeled as one sample_sheet:record (named-slot) collection, or as multiple parallel sample_sheet/list collections sharing the gene element_identifier"
    missing: "IWC corpus (115 workflows, galaxyproject/iwc, checked 2026-09-17) contains zero workflows using a record/named-slot collection type (grep -rl 'collection_type.*record' *.ga = 0 hits). The corpus's established idiom for a per-element multi-artifact tool step is N separate parallel list/sample_sheet collections that share the same element_identifier and are recombined post hoc (e.g. via collection_column_join or __FLATTEN__) -- observed directly in microbiome/mags-building/MAGs-generation.ga, where per-sample assembly/binning/QC outputs ride as several parallel collections, not one named-slot collection."
    resolved_by: freeform-summary-to-galaxy-template
    supersedes: null
    note: "Does not resolve freeform-galaxy-data-flow.md section 2 N5's open design recommendation (sample_sheet:record vs. 8 parallel collections) -- it is negative corpus evidence against the record-shaped option. Template phase should default to the corpus-precedented 8-parallel-collections-sharing-element_identifier shape unless a specific reason favors the record shape, since the record shape has no IWC exemplar to validate test/lint conventions against. [freeform-summary-to-galaxy-template, 2026-09-17] Resolved: galaxy-workflow-draft.gxwf.yml's `lexicmap_streamer_tiling_qc` step declares 8 named output ports (TODO_clean_msa_fasta, TODO_clean_haplotypes_tsv, TODO_clean_accessions_fasta, TODO_clean_expanded_accessions_fasta, TODO_flagged_accessions_fasta, TODO_flagged_tsv, TODO_cohort_ledger_tsv, TODO_summary_json), each an independent gene-keyed collection sharing the gene `element_identifier`, per this entry's recommendation -- not a `sample_sheet:record`. Topology decision made; only the wrapper's real port names remain TODO."

  - id: lexicmap-search-flatten-shape-mismatch
    status: resolved
    kind: gap
    blocking: true
    raised_by: advance-galaxy-draft-step
    step: flatten_lexicmap_results_by_gene
    unmet: "flatten_lexicmap_results_by_gene's declared behavior (__FLATTEN__, collapse a gene x index list:list down to a gene-keyed list, per its own doc and the microbiome/mags-building/MAGs-generation.ga __FLATTEN__ precedent it cites) requires a nested list:list input"
    missing: "lexicmap_search's real, pinned Tool Shed wrapper (iuc/lexicmap/lexicmap_search 0.9.0+galaxy1, changeset bcb6caec41eb, confirmed via discover-shed-tool + summarize-galaxy-tool this iteration) declares exactly one output, `out_file` (plain `data`, format `tabular`; no `discover_datasets`, no per-index/collection output). Index selection (`lexicmap_index_selection`) is bound as an in-tool-call multi-select value, not a second Galaxy-level map-over axis (per the already-settled topology commitment in ledger entry `kmindex-lexicmap-index-selection-mechanism`). Mapped over the single `gene_query_panel` axis, lexicmap_search's real output is therefore a flat `list` (one aggregate hits table per gene) -- there is no index axis for `__FLATTEN__` to collapse. draft-validate --concrete already surfaces the immediate symptom: `flatten_lexicmap_results_by_gene: step input \"input\" source \"lexicmap_search/TODO_lexicmap_hits_nested\" references unknown port\" (the port was renamed to the wrapper's real `out_file` while implementing lexicmap_search this iteration); fixing only the reference would silently paper over the deeper shape mismatch."
    resolved_by: repair-galaxy-draft-topology
    supersedes: null
    note: "Raised by advance-galaxy-draft-step (2026-09-17) while implementing lexicmap_search: confirming the real wrapper's output shape refuted the 'tool's own per-index output declaration' claim `kmindex-lexicmap-index-selection-mechanism`'s template-phase note rested on (see that entry's superseding note, same date). lexicmap_search's own step is fully concrete and correct; this entry blocked only on flatten_lexicmap_results_by_gene, which was authored as Resolved against the now-refuted list:list assumption. [repair-galaxy-draft-topology, 2026-09-17] Resolved by narrowing, not by inserting a producer: removed the now-unnecessary `flatten_lexicmap_results_by_gene` (`__FLATTEN__`) step entirely and rewired its sole consumer, `lexicmap_streamer_tiling_qc`'s `TODO_lexicmap_results` port, directly onto `lexicmap_search/out_file`. That real output already carries the exact shape (`list`, 10 elements, gene-keyed) lexicmap_streamer_tiling_qc's own doc says it expects -- no substitute producer was needed, so this is a clean bounded repair, not a workaround. Confirmed no other step or workflow output referenced `flatten_lexicmap_results_by_gene`."

artifact

test-data-refs

schema_note

Resolved workflow test inputs and expected outputs for galaxy-workflow.gxwf.yml (phix174_sra_landscape_spikein_sieve / Workflow A), derived via foundry-skills:paper-to-test-data then foundry-skills:find-test-data. Selected chain step: find-test-data (see foundry-feedback.ledger.yml phases[7].selected).

workflow

/Users/scottcain/git/dms/draft-manuscript-galaxy/galaxy-workflow.gxwf.yml

inputs

15 item(s)

gaps

4 item(s)

{
  "artifact": "test-data-refs",
  "schema_note": "Resolved workflow test inputs and expected outputs for galaxy-workflow.gxwf.yml (phix174_sra_landscape_spikein_sieve / Workflow A), derived via foundry-skills:paper-to-test-data then foundry-skills:find-test-data. Selected chain step: find-test-data (see foundry-feedback.ledger.yml phases[7].selected).",
  "workflow": "/Users/scottcain/git/dms/draft-manuscript-galaxy/galaxy-workflow.gxwf.yml",
  "scoping_decision": {
    "summary": "Full production scale (10-gene panel x 109 kmindex shards x up to 25 LexicMap indices, ~2.1M accessions) is far too large/slow for a fast workflow test. This fixture scopes the test to 2 genes, 3 kmindex shards, and 2 LexicMap indices, using real project assets where they exist and real upstream Tool-Shed test fixtures where the project's own data was never actually completed.",
    "genes_selected": [
      "E",
      "J"
    ],
    "genes_rationale": "Gene E is required to exercise the final extract_gene_e_am3_audit step (the workflow's whole reason for keeping a per-gene axis) and is the paper's flagship am3 spike-in finding. Gene J is the shortest CDS (114 bp) in the panel, kept alongside E only to give the sample_sheet collection more than one element (so map-over/collection-shape steps are genuinely exercised) while staying tiny. The other 8 genes (A, Astar, B, C, D, F, G, H, K) are dropped from this fixture, not from the production workflow.",
    "kmindex_shards_selected": [
      "GENOMIC_PHG",
      "GENOMIC_VRL",
      "METAGENOMIC_ENV"
    ],
    "kmindex_shards_rationale": "3 of the real 109 ALL_KMINDEX_DBS shard names (confirmed verbatim in /Users/scottcain/git/dms/submit_kmindex_all_genes.py) chosen for topical relevance to a phage genome search: GENOMIC_PHG (phage genomic library), GENOMIC_VRL (viral genomic), METAGENOMIC_ENV (environmental metagenome, the category the paper implicates in spike-in cross-talk).",
    "lexicmap_indices_selected": [
      "Viral",
      "BacteriaMetagenomic"
    ],
    "lexicmap_indices_rationale": "2 of the real 5-index TARGETED_PHAGE_INDICES subset (confirmed verbatim in /Users/scottcain/git/dms/pull_logan_alignments.py), out of the full 24-name ALL_LEXICMAP_INDICES list. Matches the workflow's own default (lexicmap_index_selection default already lists all 5; this fixture narrows to 2 for test speed)."
  },
  "inputs": [
    {
      "workflow_input_id": "gene_query_panel",
      "resolved": true,
      "provenance": "real_project_asset",
      "description": "Per-gene CDS query panel, scoped to 2 of the 10 real genes.",
      "elements": [
        {
          "element_identifier": "E",
          "path": "/Users/scottcain/git/dms/cds/E.fasta",
          "filetype": "fasta",
          "bytes": 311,
          "note": "273 bp CDS, coords 568..840 on NC_001422.1. Real file already used by this project's own kmindex/LexicMap submission scripts (submit_kmindex_all_genes.py, submit_remaining_logan.py)."
        },
        {
          "element_identifier": "J",
          "path": "/Users/scottcain/git/dms/cds/J.fasta",
          "filetype": "fasta",
          "bytes": 152,
          "note": "114 bp CDS, shortest gene in the panel."
        }
      ],
      "column_definitions_note": "gene_query_panel's per-element LexicMap sensitivity-override columns (align_min_match_pident, align_min_match_len, seed_min_prefix, min_qcov_per_genome) are populated in production for genes A/C/E/K only. For this fixture, set them for E (align_min_match_pident=60.0, align_min_match_len=35, seed_min_prefix=15, min_qcov_per_genome=30.0 -- real per-gene values read from /Users/scottcain/git/dms/logan_remaining_runs.json) and leave unset for J (falls back to wrapper defaults 70/50/17/unset)."
    },
    {
      "workflow_input_id": "kmindex_db_selection",
      "resolved": true,
      "provenance": "real_project_asset_names_only",
      "value": "GENOMIC_PHG,GENOMIC_VRL,METAGENOMIC_ENV",
      "note": "Real shard names (verbatim substrings of ALL_KMINDEX_DBS in submit_kmindex_all_genes.py). GAP: no real, small, prebuilt kmindex index exists anywhere for these specific Logan shard names -- the 109 shards are multi-terabyte production Logan indices hosted at usegalaxy.org and are not locally buildable/downloadable at test scale. See gaps[] below."
    },
    {
      "workflow_input_id": "kmindex_zvalue",
      "resolved": true,
      "provenance": "workflow_default",
      "value": 6
    },
    {
      "workflow_input_id": "kmindex_threshold",
      "resolved": true,
      "provenance": "workflow_default",
      "value": 0.3
    },
    {
      "workflow_input_id": "kmindex_output_format",
      "resolved": true,
      "provenance": "workflow_default",
      "value": "json"
    },
    {
      "workflow_input_id": "kmindex_fast",
      "resolved": true,
      "provenance": "workflow_default",
      "value": false
    },
    {
      "workflow_input_id": "lexicmap_index_selection",
      "resolved": true,
      "provenance": "real_project_asset_names_only",
      "value": "Viral,BacteriaMetagenomic",
      "note": "Real index names (verbatim substrings of ALL_LEXICMAP_INDICES / TARGETED_PHAGE_INDICES in pull_logan_alignments.py). GAP: same as kmindex -- these are multi-terabyte production Logan-derived LexicMap indices, not locally reproducible at test scale. See gaps[] below."
    },
    {
      "workflow_input_id": "lexicmap_top_n_genomes",
      "resolved": true,
      "provenance": "workflow_default",
      "value": 0
    },
    {
      "workflow_input_id": "lexicmap_advanced_all",
      "resolved": true,
      "provenance": "workflow_default",
      "value": true
    },
    {
      "workflow_input_id": "tiling_qc_min_coverage",
      "resolved": true,
      "provenance": "workflow_default",
      "value": 0.8
    },
    {
      "workflow_input_id": "tiling_qc_min_coverage_partial",
      "resolved": true,
      "provenance": "workflow_default",
      "value": 0.5
    },
    {
      "workflow_input_id": "tiling_qc_min_pident",
      "resolved": true,
      "provenance": "workflow_default",
      "value": 60
    },
    {
      "workflow_input_id": "tiling_qc_max_internal_stops",
      "resolved": true,
      "provenance": "workflow_default",
      "value": 0
    },
    {
      "workflow_input_id": "tiling_qc_sample_cap",
      "resolved": true,
      "provenance": "workflow_default",
      "value": 10
    },
    {
      "workflow_input_id": "tiling_qc_allow_frameshifts",
      "resolved": true,
      "provenance": "confirmed_from_real_source_(open-requirements ledger entry allow-frameshifts-default-unconfirmed, resolved)",
      "value": false,
      "note": "Confirmed OFF/False by direct read of nekrut/disassembler python/lexicmap_streamer.py argparse (store_true, no default=True)."
    }
  ],
  "tool_level_structural_fixtures": {
    "note": "Because no small real Logan-shard kmindex DB or Logan-derived LexicMap index exists anywhere (in this repo or upstream), the two real external Galaxy tools in this workflow (kmindex_query 0.6.1+galaxy4, lexicmap_search 0.9.0+galaxy1) can only be exercised end-to-end in THIS test using their own upstream Tool Shed functional-test fixtures -- real, already-committed, already-used-for-CI assets, but generic smoke-test data (synthetic repeat sequences / two small real RefSeq viral genomes), NOT phiX174 biology and NOT the named Logan shards/indices above. Recommended use: a structural/wiring-only sub-test of steps kmindex_containment_screen and lexicmap_search in isolation, separate from a full end-to-end run.",
    "kmindex_query": {
      "source": "galaxyproject/tools-iuc, tools/kmindex/test-data (main branch)",
      "closest_analog_test": "kmindex_query.xml <test> #6 ('using register index, JSON output'): db_opts_selector=db, kmindex=register (expands to index1,index2 -- a real 2-shard multi-select, structurally identical to this workflow's db_opts|kmindex multi-select binding) -> output_json list collection with 2 elements (index1, index2).",
      "files": {
        "query_fasta": "https://raw.githubusercontent.com/galaxyproject/tools-iuc/main/tools/kmindex/test-data/query1.fasta",
        "prebuilt_index_register": "https://github.com/galaxyproject/tools-iuc/tree/main/tools/kmindex/test-data/register_index",
        "prebuilt_index1": "https://github.com/galaxyproject/tools-iuc/tree/main/tools/kmindex/test-data/index1",
        "prebuilt_index2": "https://github.com/galaxyproject/tools-iuc/tree/main/tools/kmindex/test-data/index2",
        "expected_output_index1": "https://raw.githubusercontent.com/galaxyproject/tools-iuc/main/tools/kmindex/test-data/expected_query2_index1.json",
        "expected_output_index2": "https://raw.githubusercontent.com/galaxyproject/tools-iuc/main/tools/kmindex/test-data/expected_query2_index2.json"
      },
      "caveat": "tools-iuc's own kmindex.loc carries the comment: 'This file is just a placeholder since Galaxy does not yet support uploading a kmindex index, which is required for functional tests.' The indexed sequences (seq1/seq2/seq3, synthetic ATCG/CGCG/TATA/GAGA repeats) have no relationship to phiX174 or the named Logan shards."
    },
    "lexicmap_search": {
      "source": "galaxyproject/tools-iuc, tools/lexicmap/test-data (main branch)",
      "closest_analog_test": "lexicmap.xml <test> #3/#4/#6 (query one/multi FASTA against LexicMapIndexCombined = db.lmi + db2.lmi, i.e. a real 2-index multi-select, structurally identical to this workflow's db_opts|lexicmap_index multi-select binding).",
      "real_indexed_genomes": [
        "GCF_001502155.1 (ViralProj307776)",
        "GCF_001502175.1 (ViralProj307780)"
      ],
      "files": {
        "prebuilt_index_1": "https://github.com/galaxyproject/tools-iuc/tree/main/tools/lexicmap/test-data/db.lmi",
        "prebuilt_index_2": "https://github.com/galaxyproject/tools-iuc/tree/main/tools/lexicmap/test-data/db2.lmi",
        "query_fasta": "https://raw.githubusercontent.com/galaxyproject/tools-iuc/main/tools/lexicmap/test-data/lexicmap_query3.fasta",
        "expected_output_with_all_columns_schema": "https://raw.githubusercontent.com/galaxyproject/tools-iuc/main/tools/lexicmap/test-data/lexicmap_query_result3.tsv"
      },
      "column_schema_confirmed": "With --all (this workflow's lexicmap_advanced_all=true), lexicmap.xml's column_names metadata action declares: query,qlen,hits,sgenome,sseqid,qcovGnm,cls,hsp,qcovHSP,alenHSP,pident,gaps,qstart,qend,sstart,send,sstr,slen,evalue,bitscore,cigar,qseq,sseq,align -- confirms lexicmap_streamer.py's required columns (sgenome, pident, qstart, qend, qseq, sseq) are all genuinely present in real lexicmap_search output when --all is set.",
      "caveat": "lexicmap_index.loc carries the same 'placeholder ... Galaxy does not yet support uploading a lexicmap index' comment. The 2 indexed genomes are real small viral RefSeq assemblies but are not phiX174 and not Logan-derived domain indices."
    }
  },
  "known_accessions_for_biological_grounding": {
    "positive_controls_real": {
      "source": "BioProject PRJNA1174868 / SRP539859 (Idaho 2024 chemostat time series), cited in freeform-summary.md Stage E and phix174_sra_evolution_manifest.tsv",
      "accessions": [
        "SRR31059334",
        "SRR31059335"
      ],
      "note": "Real SRA runs known to derive from a laboratory phiX174 chemostat competitive-growth experiment (guaranteed genuine ΦX174-positive reads, tracking Gene G specifically per the paper). NOT yet tied to any actual recorded kmindex containment score or LexicMap hit for genes E/J in this repo -- no completed kmindex/LexicMap run against these accessions exists (kmindex_runs.json / logan_remaining_runs.json both show status: 'queued', never completed)."
    },
    "negative_control": {
      "resolved": true,
      "accession": "SRR40691913",
      "source": "User-supplied (2026-09-17), attributed to \"a reliable source\" as expected to have zero phiX174 gene-panel containment/alignment. Not independently derived from freeform-summary.md or any file in this repo.",
      "verification_status": "unverified_by_this_pipeline: no kmindex/LexicMap run has actually been executed against this accession in this run. Treat as a candidate true-negative to be confirmed empirically (e.g. via kmindex_containment_screen + lexicmap_search) before relying on it as a hard assertion target; do not assume zero-hit status is proven until that run completes.",
      "reason_previously_unresolved": "freeform-summary.md's nominal-taxonomy audit (Stage D) catalogs BioProjects that ARE phiX174-tagged (misfiled or genuine), not a verified phiX174-negative accession. No accession in this repo or the paper was explicitly documented as a true negative for phiX174 gene-panel containment/alignment, hence the original gap -- now resolved via direct user input rather than repo/paper evidence."
    }
  },
  "expected_output_ground_truth_full_scale_only": {
    "note": "These are REAL, already-computed full-production-scale numbers from this project's own results/ directory -- NOT achievable by the tiny scoped fixture above (2 genes x 3 shards x 2 indices vs. the real 10 genes x 109 shards x up to 25 indices x ~2.1M accessions). Useful only as directional/sanity-bound references for a full-scale run, not as exact-match assertions for the fast test.",
    "ingestion_manifest_rows": {
      "source": "/Users/scottcain/git/dms/results/00_ingestion/lexicmap_9genes_ingestion_manifest.csv",
      "gene_J": "length_bp=114, total_accessions_evaluated=2278534, clean_full_length=2184554, flagged_internal_stops=63, clean_unique_haplotypes=1081",
      "gene_E": "length_bp=273, total_accessions_evaluated=2392457, clean_full_length=52862, flagged_internal_stops=2215172, clean_unique_haplotypes=1379"
    },
    "gene_e_am3_audit": {
      "source": "/Users/scottcain/git/dms/results/02_layer1_quasispecies/gene_e_am3_quarantine_audit.json",
      "total_evaluated_accessions": 2392457,
      "clean_wildtype_accessions": 52862,
      "am3_flagged_stop_accessions": 2215172,
      "am3_stop_fraction_pct": 92.59,
      "am3_genomic_coordinate": 587,
      "am3_mutation_gene_e": "TGG->TAG (Trp7Stop)"
    }
  },
  "gaps": [
    {
      "id": "no-real-small-kmindex-logan-shard-subset",
      "unmet": "A small, real, downloadable subset of any actual named Logan kmindex shard (GENOMIC_PHG, GENOMIC_VRL, METAGENOMIC_ENV, or any of the other 106) containing verifiable phiX174-related hits.",
      "why": "The 109 Logan shards are production-scale (multi-TB) indices hosted only at usegalaxy.org via a Tool Shed data-table entry (db_opts|kmindex); kmindex itself has no public 'build a tiny shard from a public FASTA subset' precedent in this repo or upstream tools-iuc, and this project's own kmindex run (kmindex_runs.json) never completed (status: queued).",
      "what_would_resolve_it": "Either (a) the user grants access to a live usegalaxy.org history/API key so the workflow's kmindex_containment_screen step can be run for real against a couple of the smallest real shards, or (b) the user (or a downstream build step) constructs a tiny local kmindex index via kmindex_query's own kmindex_build.xml wrapper, seeded with cds/E.fasta plus a handful of real decoy sequences, named to mimic one real shard for structural purposes only."
    },
    {
      "id": "no-real-small-lexicmap-logan-index-subset",
      "unmet": "A small, real, downloadable subset of any actual named LexicMap Logan domain index (Viral, BacteriaMetagenomic, or any of the other 22) with verifiable phiX174 hits.",
      "why": "Same production-scale/hosting constraint as kmindex above; this project's own LexicMap runs also never completed (logan_remaining_runs.json / logan_structural_runs.json both show status: 'queued').",
      "what_would_resolve_it": "Same two options as above, using lexicmap-index.xml to build a tiny local index from cds/*.fasta plus decoys."
    },
    {
      "id": "no-real-completed-lexicmap-hits-table-for-lexicmap-streamer",
      "unmet": "A real LexicMap results TSV (the lexicmap_streamer_tiling_qc step's second required input, -f/--file) containing genuine phiX174 Gene E / Gene J hits, ideally including at least one wildtype-clean row and one am3-like premature-stop row.",
      "why": "No such file exists anywhere in this repo (msas/, msas_sub/, logan_alignments/ contain post-hoc EVcouplings-style protein/codon MSAs for HyphAeon, not raw per-accession LexicMap tabular hits; results/ has only already-aggregated summary CSVs/JSONs, not the underlying hit table).",
      "what_would_resolve_it": "implement-galaxy-workflow-test should hand-construct a small synthetic TSV using the REAL, confirmed column schema in tool_level_structural_fixtures.lexicmap_search.column_schema_confirmed (sgenome, pident, qcovHSP, qstart, qend, qseq, sseq, etc.), with qseq/sseq values derived from real cds/E.fasta and cds/J.fasta sequences with hand-introduced positive/negative cases -- clearly labeled as synthetic, not as a real accession's genuine sequencing data."
    },
    {
      "id": "no-verified-phix174-negative-control-accession",
      "unmet": "See known_accessions_for_biological_grounding.negative_control above.",
      "why": "freeform-summary.md and this repo document only phiX174-tagged BioProjects; none is documented as a confirmed non-hit.",
      "what_would_resolve_it": "User input: name a specific SRA accession known/expected to have zero phiX174 gene-panel containment, for use as a true-negative test case.",
      "status": "resolved_by_user_input",
      "resolution": "User supplied SRR40691913 (2026-09-17) as the expected negative control. Still needs an actual kmindex/LexicMap run to empirically confirm zero containment before it's used as a hard assertion in the test plan -- see known_accessions_for_biological_grounding.negative_control.verification_status."
    }
  ]
}

• Workflow debug report — galaxy-workflow.gxwf.yml
• Test 1 — `kmindex_wiring_smoke_generic_fixtures` (test_index 1)
• Test 2 — `gene_e_j_am3_diagnostic_synthetic_lexicmap_index` (test_index 2)
• Evidence
• Feedback ledger
• Overall assessment of the deliverable

# Workflow debug report — galaxy-workflow.gxwf.yml

Run: draft-manuscript-galaxy (phase 12 of PAPER → GALAXY). Inputs: `workflow-test-result.json`,
`planemo-test1-output.{json,html,log}`, `planemo-test2-output.{json,html,log}`.

## Test 1 — `kmindex_wiring_smoke_generic_fixtures` (test_index 1)

**Classification: staging failure, external defect (galaxy-tool-util / Planemo), not a
workflow-authoring defect.** No Galaxy invocation was ever created.

Evidence (`planemo-test1-output.json`, `planemo-test1.log:1782-1812`):

- `execution_problem: "'rows'"`, `status: "error"`, `invocation_details: null`, `job: null` —
  the failure happened before any invocation/job object existed.
- Exact traceback:
  `planemo/galaxy/activity.py:441 stage_in` → `galaxy/tool_util/client/staging.py:275 stage` →
  `galaxy/tool_util/cwl/util.py:388 galactic_job_json` → `:234 replacement_item` →
  `:362 replacement_collection` → `KeyError: 'rows'`, i.e.
  `kwds["rows"] = value["rows"]` unconditionally indexes an optional key.
- The test case's `gene_query_panel` input in `planemo-test1-output.json` is a
  `collection_type: sample_sheet` Collection with `elements` but **no `rows` key at all** —
  and this exact test file previously passed `gxwf validate-tests --workflow ... --json`
  cleanly (per `workflow-test-result.json`'s `static_validation.valid: true`), so the schema
  legitimately treats `rows` as optional.
- Contrast: test 2's `gene_query_panel` *does* carry a `rows:` block and staged past this exact
  code path without error, isolating the variable to the missing key, not a general staging bug.

This is galaxy-tool-util 25.1.2 (vendored in planemo 0.75.47), a related project, not this
workflow. It is already tracked in the feedback ledger as
`galaxy-tool-util-replacement-collection-requires-rows-key-unconditionally` (status: open).

**Recommended next step:** upstream bug report against `galaxy.tool_util.cwl.util.replacement_collection`
(use `.get("rows", {})` instead of `value["rows"]`). As a practical local workaround (not a
schema violation, just filling in an optional field defensively) the test's `sample_sheet` job
input could add an empty `rows: {}`/`rows: []` block to unblock staging without waiting on the
upstream fix — but this is an environment workaround, not something to silently "fix" in the
workflow's own logic. Route to: user/future run (apply workaround) + upstream `galaxyproject/galaxy` issue.

## Test 2 — `gene_e_j_am3_diagnostic_synthetic_lexicmap_index` (test_index 2)

**Classification: workflow-invocation refusal at request time (HTTP 400, "required tools are
not installed") — evidence points to a toolbox-state/timing artifact as the primary cause,
compounded by one genuine, platform-specific dependency-resolution failure that is a real but
separate blocker.**

### Evidence

`planemo-test2-output.json`: `execution_problem` is a `bioblend.ConnectionError: Unexpected
HTTP status code: 400` on `POST /api/workflows/961e7c4742a92de4/invocations`
(`planemo-test2.log:1775`), body:

```
err_msg: "Workflow was not invoked; the following required tools are not installed:
toolshed.g2.bx.psu.edu/repos/nml/collapse_collections/collapse_dataset (version 5.1.0),
toolshed.g2.bx.psu.edu/repos/iuc/lexicmap/lexicmap_search (version 0.9.0+galaxy1),
toolshed.g2.bx.psu.edu/repos/iuc/kmindex/kmindex_query (version 0.6.1+galaxy4),
lexicmap_streamer (version 1.0.0), flatten_gene_summary_json_to_row (version 1.0.0),
kmindex_hit_dedup_max_score (version 1.0.0),
toolshed.g2.bx.psu.edu/repos/iuc/collection_column_join/collection_column_join (version 0.0.3)"
```

`invocation_details: null`, `job: null` — this is a request-time (pre-invocation) rejection per
galaxy-workflow-invocation-failure-reference's "Request-time validation" surface; no invocation
object was ever created, so invocation-state/message fields don't apply here.

**The key finding** (`planemo-test2.log:292`):

```
galaxy.tool_shed.galaxy_install.repository_dependencies.repository_dependency_manager INFO
11:50:04,798 Skipping installation of revision 90981f86000f of repository 'collapse_collections'
because it was installed with the (possibly updated) revision 90981f86000f and its current
installation status is 'Installed'.
```

`collapse_dataset` was **already fully installed from a cached prior run and its install was
skipped entirely** — no fresh clone, no fresh dependency resolution — yet it is *still* listed
in the 400 as "not installed" 54 seconds later. A tool with a confirmed-good, pre-existing
"Installed" status being rejected rules out "genuine dependency failure" as the explanation for
*that* tool, and is the strongest evidence available that the invocation-time tool-availability
check is reading stale/incomplete toolbox state rather than reality — i.e., a toolbox-reload
race, consistent with all 7 required tools (4 real + 3 UDT) being rejected as one uniform block
regardless of each one's actual install state.

Timing: the last `reload_toolbox` control-task cycle logged completes at `11:50:13,808` (for
`collection_column_join`, cloned `11:50:10,458`); the invocation POST fires 45s later at
`11:50:58,531`. Toolbox reload dispatch is not obviously still in flight by then — so the race,
if real, is more likely in how the invocation-validation code path reads toolbox state (e.g. a
cached toolbox snapshot on the request-handling worker) than in reload latency itself.

**Separately, a real dependency-resolution failure was also observed** — 8 repeated occurrences
of (`planemo-test2.log:313-520`, no timestamps on the subprocess output but bounded between
`11:50:13,808` and `11:50:43,834`):

```
Platform: osx-arm64
Solving environment: ...working... failed
PackagesNotFoundError: The following packages are not available from current channels:
  - coreutils=8.25
```

Web search confirms this is a real, structural platform limitation, not transient: the
`coreutils=8.25` pin is an old build that conda-forge never shipped for `osx-arm64` (current/
newer `coreutils` builds do support `osx-arm64`; older exact-pinned versions like 8.25 are
`linux-64`/`osx-64`-only). This is an Apple Silicon macOS host limitation on whichever Tool Shed
wrapper(s) pin that old `coreutils` requirement — it cannot be "coded around" in the workflow;
it needs either a different install host/architecture (e.g. Rosetta/`osx-64` conda subdir), or
the wrapper's own `coreutils` requirement pin needs to be loosened upstream.

Because `collapse_dataset`'s dependencies were never touched this run (skip-install path) while
the coreutils failures occurred elsewhere in the timeline, the two problems are best read as
distinct: (1) a toolbox-availability/timing defect affecting the invocation-validation check for
all 7 tools uniformly, and (2) a real, platform-specific conda resolution failure for at least
one of the freshly-installed shed wrappers, which would block that job even if (1) were fixed.
The 3 UDTs remain independently blocked regardless of either issue — they were never in any
toolbox (`tool_dir` scanner limitation, unchanged from phase 11's finding).

**Recommended next steps:**
- UDT-loading gap (structural, unresolved): route to a future Foundry run/maintainer decision —
  `author-galaxy-tool-wrapper` or the harness needs a lowering path from `GalaxyUserTool` YAML to
  classic Galaxy tool XML (or another Planemo-supported loading mechanism) before this workflow's
  3 authored tools can ever be tested locally. Tracked: `run-workflow-test-no-mechanism-to-install-authored-galaxyusertool-udts`.
- Toolbox-reload/timing race: investigate Galaxy's `/api/tool_shed_repositories` install-status
  and the invocation-validation code path's toolbox source (cached vs. live) — a future
  `run-workflow-test`/Planemo run, not something this workflow's authors can fix. New evidence
  captured this phase (see ledger entry added below).
- `coreutils=8.25`/osx-arm64: flag to the user as an environment/platform constraint, not a code
  fix — either re-run on `osx-64`/Rosetta emulation or an x86_64 CI host, or wait for/request an
  upstream wrapper update that loosens the `coreutils` pin.
- `lexicmap_index_selection` placeholder (`"PhiX174E_J_Am3ToyIndex"`): pre-existing, documented
  deferral from `galaxy-test-plan.yml`; unreached this run because tool-install failed first —
  still the next real blocker once 1-3 above are resolved.

## Feedback ledger

Reviewed `foundry-feedback.ledger.yml` (16 prior entries). The two entries phase 11 already
filed for this run's UDT-loading gap and the `rows` KeyError already carry strong, specific
evidence and did not need re-filing or amendment.

One new entry appended (new evidence this phase, not previously captured): the collapse_dataset
"skip-install, already Installed, still rejected" finding, which is a concrete diagnostic signal
that `galaxy-workflow-invocation-failure-reference.md` doesn't currently document — id
`galaxy-workflow-invocation-check-rejects-previously-installed-tool-as-not-installed` (kind:
gap, subject: research, `galaxy-workflow-invocation-failure-reference.md`, severity: major).

## Overall assessment of the deliverable

`galaxy-workflow.gxwf.yml` + `galaxy-workflow.gxwf-tests.yml` passed static/structural
validation cleanly (`gxwf validate-tests ... --json` → `valid: true`, no errors) and phase 10's
`galaxy-workflow-validation-result.json` was clean going into testing. Neither test failure this
phase points at a wiring, collection-shape, or authoring defect in the workflow itself:

- Test 1 died in Planemo's own staging code on a legitimately-optional field — an upstream bug.
- Test 2 never got a job to run; it was refused at the API gate by what most of the evidence
  points to as toolbox/tooling infrastructure state, not the workflow's tool references (all of
  which are correctly pinned, real Tool Shed ids/versions per `galaxy-tool-pin.json`).

The workflow is structurally sound and blocked on tooling/environment gaps, not authoring
defects: no mechanism exists yet to load the 3 hand-authored UDTs into a Planemo Galaxy
toolbox, the toolbox-availability check at invocation time appears unreliable/stale in this
harness's environment, and one real Tool Shed wrapper hits a hard osx-arm64 conda-dependency
wall. None of these are fixable by further editing `galaxy-workflow.gxwf.yml` itself. The one
remaining known, already-documented workflow-level gap (`lexicmap_index_selection` placeholder)
is a legitimate, previously-deferred TODO, not a defect exposed by this run.

{
  "workflow_path": "galaxy-workflow.gxwf.yml",
  "test_file_path": "galaxy-workflow.gxwf-tests.yml",
  "status": "fail",
  "not_run_reason": null,
  "static_validation": {
    "command": "gxwf validate-tests galaxy-workflow.gxwf-tests.yml --workflow galaxy-workflow.gxwf.yml --json",
    "valid": true,
    "errors": []
  },
  "galaxy_mode": {
    "kind": "planemo-managed",
    "install_flags": [
      "--install_galaxy"
    ],
    "galaxy_branch_pinned": false,
    "galaxy_checkout": "~/.planemo/gx_repo (reused cached checkout from phase 9, HEAD 18bb36a9f49071da5e502b162c9a6905d7b78aae, 2026-09-17)",
    "galaxy_venv": "~/.planemo/gx_venv_3 (Python 3.13.5, reused cached venv)",
    "udt_supply_attempt": {
      "mechanism_tried": "planemo test --extra_tools <dir containing the 3 GalaxyUserTool YAML files>",
      "outcome": "did not register any of the 3 UDTs",
      "evidence": "planemo wrote `<tool_dir dir=\".../udt-tools\" />` into the generated tool_conf.xml (confirmed by reading the file directly from the live Planemo temp config dir); Galaxy's toolbox parsed that tool_conf.xml but zero log lines in either run's planemo log reference any of the 3 UDT ids/filenames or `GalaxyUserTool`. The subsequent workflow-invocation HTTP 400 for test case 2 explicitly listed `lexicmap_streamer (version 1.0.0)`, `flatten_gene_summary_json_to_row (version 1.0.0)`, and `kmindex_hit_dedup_max_score (version 1.0.0)` among tools 'not installed', confirming the toolbox never registered them.",
      "root_cause": "Galaxy's classic `tool_dir` toolbox scanner auto-discovers XML tool wrappers only. A GalaxyUserTool YAML document (per author-galaxy-tool-wrapper's own SKILL.md: 'a single GalaxyUserTool YAML document, not Galaxy XML') has no lowering step to XML documented anywhere in this Foundry's packaged references, no `gxwf` subcommand performs this conversion, and no other planemo flag was found that accepts this format directly.",
      "classification": "genuinely unresolved infrastructural gap, not a transient install failure -- logged as feedback ledger entry run-workflow-test-no-mechanism-to-install-authored-galaxyusertool-udts"
    }
  },
  "test_cases": [
    {
      "id": "galaxy-workflow.gxwf.yml_0",
      "name": "kmindex_wiring_smoke_generic_fixtures",
      "test_index": 1,
      "planemo_command": "planemo test --install_galaxy --test_index 1 --extra_tools <udt-dir> --test_output_json planemo-test1-output.json --test_output planemo-test1-output.html galaxy-workflow.gxwf.yml",
      "status": "fail",
      "modality": "staging failure (client-side, pre-invocation)",
      "planemo_result": {
        "execution_problem": "'rows'",
        "status": "error",
        "invocation_details": null,
        "job": null,
        "output_problems": []
      },
      "root_cause": "Uncaught KeyError: 'rows' in galaxy.tool_util.cwl.util.replacement_collection (galaxy-tool-util==25.1.2, vendored into planemo==0.75.47). That function unconditionally executes `kwds[\"rows\"] = value[\"rows\"]` for any collection_type starting with 'sample_sheet'. This test case's gene_query_panel Collection intentionally omits `rows:` (not meaningful for generic non-phiX174 fixture content, per the test file's own comment), which tests-format.schema.json's Collection $def treats as optional -- and this exact test file passed gxwf validate-tests --workflow cleanly with rows absent. No Galaxy invocation was ever created; no tool (kmindex, lexicmap, collapse_dataset, or any UDT) was exercised by this test case.",
      "traceback_tail": "planemo/galaxy/activity.py:441 stage_in -> galaxy/tool_util/client/staging.py:275 stage -> galaxy/tool_util/cwl/util.py:388 galactic_job_json -> :234 replacement_item -> :362 replacement_collection -> KeyError: 'rows'",
      "artifacts": {
        "test_output_json": "planemo-test1-output.json",
        "test_output_html": "planemo-test1-output.html",
        "raw_log": "planemo-test1.log"
      },
      "next_reference_surface": "galaxy.tool_util.cwl.util.replacement_collection source (galaxy_tool_util-25.1.2, vendored in planemo's venv) -- fix or work around the unconditional `value[\"rows\"]` access before this test case can stage at all; see feedback ledger entry galaxy-tool-util-replacement-collection-requires-rows-key-unconditionally"
    },
    {
      "id": "galaxy-workflow.gxwf.yml_1",
      "name": "gene_e_j_am3_diagnostic_synthetic_lexicmap_index",
      "test_index": 2,
      "planemo_command": "planemo test --install_galaxy --test_index 2 --extra_tools <udt-dir> --test_timeout 600 --test_output_json planemo-test2-output.json --test_output planemo-test2-output.html galaxy-workflow.gxwf.yml",
      "status": "fail",
      "modality": "tool-install failure (workflow invocation refused by Galaxy before any job ran)",
      "planemo_result": {
        "execution_problem": "Unexpected HTTP status code: 400: {\"err_msg\":\"Workflow was not invoked; the following required tools are not installed: toolshed.g2.bx.psu.edu/repos/nml/collapse_collections/collapse_dataset (version 5.1.0), toolshed.g2.bx.psu.edu/repos/iuc/lexicmap/lexicmap_search (version 0.9.0+galaxy1), toolshed.g2.bx.psu.edu/repos/iuc/kmindex/kmindex_query (version 0.6.1+galaxy4), lexicmap_streamer (version 1.0.0), flatten_gene_summary_json_to_row (version 1.0.0), kmindex_hit_dedup_max_score (version 1.0.0), toolshed.g2.bx.psu.edu/repos/iuc/collection_column_join/collection_column_join (version 0.0.3)\",\"err_code\":0}",
        "status": "error",
        "invocation_details": null,
        "job": null,
        "output_problems": []
      },
      "root_cause_notes": [
        "The 3 authored UDTs (lexicmap_streamer, flatten_gene_summary_json_to_row, kmindex_hit_dedup_max_score) were never in the toolbox at all -- consistent with test case 1's finding and the confirmed absence of any UDT-related toolbox log lines (see udt_supply_attempt above). This part is a confirmed, structural block.",
        "The real Tool Shed tools (collapse_dataset, lexicmap_search, kmindex_query, collection_column_join) were also reported not-installed, despite install_manager log lines showing 'Cloning repository' for all four (collapse_collections was even logged as already 'Installed' from a cached prior run). Only ~15-48s elapsed between each clone and the invocation attempt. Given a `Solving environment: ...working... failed / PackagesNotFoundError: coreutils=8.25` conda resolution failure was observed for at least one dependency shortly before the invocation attempt (coreutils=8.25 predates osx-arm64 conda-forge/bioconda builds -- a plausible platform-specific limitation of this macOS Apple Silicon host, not necessarily a Galaxy/Foundry defect), this looks like either (a) a genuine dependency-resolution failure for these wrappers on this platform, or (b) a race where planemo attempted the invocation before the async toolbox reload/install-completion signal caught up. This phase did not have time/scope to distinguish (a) from (b); it is deliberately left as the concrete next-step question rather than guessed at."
      ],
      "artifacts": {
        "test_output_json": "planemo-test2-output.json",
        "test_output_html": "planemo-test2-output.html",
        "raw_log": "planemo-test2.log"
      },
      "next_reference_surface": "Galaxy's /api/tool_shed_repositories install-status API and the per-repository dependency-install logs (job_working_directory / conda build logs) for collapse_collections, kmindex, lexicmap, and collection_column_join on this host's osx-arm64 platform -- to determine whether the coreutils=8.25 (or similar) conda resolution genuinely failed for these wrappers versus a toolbox-reload timing race; galaxy-workflow-invocation-failure-reference.md's install/dependency guidance and the already-open feedback entry tool-util-cli-toolshed-fetch-rejects-real-filtered-list-collection-output are related but do not by themselves resolve this.",
      "known_preexisting_deferral": "This test case's lexicmap_index_selection is a documented placeholder (\"PhiX174E_J_Am3ToyIndex\", galaxy-test-plan.yml unresolved[0]/[1]); even if tool-install were resolved, this case was already expected by its own authors to fail/stall at lexicmap_search pending a real toy LexicMap index. That downstream gap was not reached this run because tool-install failed first."
    }
  ],
  "feedback": {
    "entries_appended": [
      "run-workflow-test-no-mechanism-to-install-authored-galaxyusertool-udts",
      "galaxy-tool-util-replacement-collection-requires-rows-key-unconditionally"
    ],
    "ledger_path": "foundry-feedback.ledger.yml"
  },
  "handoff_to_phase_12": {
    "recommended_focus_order": [
      "1. Decide whether to fix/work around galaxy-tool-util's replacement_collection 'rows' KeyError (blocks test case 1 from staging at all) -- this is the cheapest unblock and does not depend on tool installation.",
      "2. Resolve the UDT-installation gap: either author an XML lowering of the 3 GalaxyUserTool definitions for toolbox loading, or find/confirm a genuine Planemo/Galaxy mechanism for YAML-defined tools (none confirmed working in this run).",
      "3. Re-investigate why the 3 real Tool Shed tools also reported not-installed at invocation time on this host/platform -- rule out a conda dependency-resolution failure (coreutils=8.25 on osx-arm64) vs. a toolbox-reload race before concluding tool-install itself is broken.",
      "4. Only after 1-3, the still-open lexicmap_index_selection placeholder (test case 2) becomes the next real blocker."
    ]
  }
}