Ingesting new samples into the hop atlas: runbooks by data type

Date: 2026-10-08 (updated 2026-10-13: Carey 2026 live, Zhang 2017 opt-in, per-reference calls, manifests). Status: reconstructed from the project's dated records (sync/, sync_archive/, annevo_eval/docs/, cluster/, /data/hop_data/*/scripts, memory, git). It describes what was actually run for each cohort, with the file each step rests on. None of these runbooks has been executed end to end; each one marks inferred steps (inferred) and missing ones GAP.

Data type	Runbook	Cohorts it was reconstructed from	Has reached the atlas
GBS, three kinds: ApeKI SE, ApeKI PE, PstI+ApeKI double digest PE	`GBS.md`	Havill, diversity panel, USDA, OSU (ApeKI SE); Wye (ApeKI PE); Cascade (PstI+ApeKI)	introgression base + layers + sex + phasing; dosage for Havill, Cascade, Wye
Short-read WGS	`WGS.md`	Apollo×Pub_740M, KR (cordifolius), Cascade/Wye parents, Carey 2026	Apollo, KR, parents; Carey 2026 live since 2026-10-09 (base + layers + sex; manifest-driven stage scripts `cohorts/carey2026/pipeline/`)
GRAS-Di	`GRASDI.md`	Saaz×HL1012 (PRJDB19078, 257 runs)	introgression base (route V) + subspecies + sex + phasing; dosage
RNA-seq	`RNASEQ.md`	Wye leaf RNA (PRJNA694508)	Wye RNA lane of the introgression atlas only
Genome assemblies (own-built and external)	`ASSEMBLY.md`	Apollo phased, Cascade per-homolog, Saaz/HL1012, drHumLupu, 21110M, Akagi, Dovetail Cascade, Carey	ribbon atlas rows; alignment/dosage references; pangenome graphs; diagnostic catalogue

Genotype-only inputs (DArT Pilgrim, VCF-only panels, the Zhang 2017 HapMap, served as an opt-in dataset since 2026-10-08) are covered in ../ADDING_DATA.md and the GBS runbook §1.

Per-reference calls (the atlas Reference selector) are an extra stage after S5 for cohorts that should be shown on other assemblies: BAMs per reference → annevo_eval/methods/per_frame_ancestry/ → hop_atlas/tools/build_frames.py (COHORTS) + patch_frames.py → deploy (../components/INTROGRESSION.md §8e). Done for Zenith x 21058M; other cohorts in progress.

The common skeleton

Every read-based data type goes through the same stages; only the settings differ. "Generic" = runnable for a new cohort by configuration; "copy" = a cohort-specific script you copy and edit.

Stage	GBS	WGS	GRAS-Di	RNA-seq	Generic today?
S0 cohort manifest (runs, roles, parents, truth plants, exclusions, assay)	§1	S0	Stage 0	Stage 0	No (hard-coded in ≥ 15 scripts)
S1 download → PBSS, integrity gates	Stage 1	S1	Stage 1	Stage 1	copy (best: OSU `fetch_one.sh`, Carey Lepton workers)
S2 read QC (enzyme/barcode, platform from headers, identity)	Stage 2	S2	Stage 2	Stage 2, 3c	partly (`paternity_check.py` generic)
S3 alignment to 21110M (`pbrun`, assay dup rule)	Stage 3	S3	Stage 3	Stage 3a–b (STAR)	yes: `cluster/lepton_linear` (config per cohort)
S4 allele counts at the diagnostic catalogue (route B)	Stage 4	S4	Stage 4	Stage 4	nearly (`m01_count_one.sh`); job tables hand-written
S5 NA/EU calls (EM fit) + evaluation	Stage 5	S5	Stage 5	Stage 5–6	copy (`o03`/`o04`/`o05`)
S6 base viewer export	Stage 5	S6	Stage 5	Stage 7	copy (`o07`/`o08`/`o17`, `o15`)
S7 layers: subspecies/lineage, sex lane, phasing, pedigree	Stage 6	S7	Stage 6–7	none done	export/test generic (`atlas_layers.py`); counts and cohort lists copy
S8 dosage atlas	Stage 7	S8	Stage 8	none done	copy (~6 hard-coded lists)
S9 stage → test → deploy	Stage 8	S9	Stage 9	Stage 8	yes: `atlas_layers.py`, `atlas_deploy.py`
S10 record (PIPELINE.sh, `register_product.sh`, `catalog.yaml`, CHANGELOG)	Stage 8	S9	Stage 9	Stage 8	yes (rules)

Assemblies follow a different chain (assemble/obtain → QC → naming + orientation → ribbon row → reference registration → graph → catalogue impact); see ASSEMBLY.md §2.

Rules that hold for every data type (each from a recorded failure)

Duplicate marking by assay. GBS and GRAS-Di: never (pbrun --no-markdups; counts with --ff UNMAP,SECONDARY,QCFAIL). WGS: mark and drop (--ff …,DUP). Gate: samtools view -c -f 1024 is 0 for GBS/GRAS-Di, > 0 for WGS (ASSAYS.md §1.1; cluster/lepton_linear/README.md). The generic aligner defaults parents to dup-marking: set PARENT_MARKDUP=0 when parents are GBS/GRAS-Di.
Integrity = md5 + published size or read count, never gzip -t alone (it missed 10 truncated Apollo runs; ASSAYS.md §4.2j). Decode platform and read length from read headers, not ENA labels.
Base calls from BAM pileups (route B), not from variants-only VCFs (route V). Variants-only VCFs hide reference-allele evidence (ASSAYS.md 2026-09-30). Crosses moved to route B; Havill, divpanel, USDA and Saaz are still served from route V (see decisions below).
One alignment per cohort feeds every layer. Several cohorts were aligned more than once and different layers read different BAMs (GBS §4 item 5, GRAS-Di G10). Not yet stated as a rule anywhere; recorded here as the convention to follow, pending PI confirmation.
Keep the BAMs on PBSS (PUSH_SUBS must include bam); USDA BAMs exist only on Lepton fs1.
Same plant, different assay: never merge rows. Each library is its own atlas row <cohort>|<sample>; identity is a same-plant group in annevo_eval/sample_identity/ (WGS §4). Cross-assay identity thresholds were calibrated on GBS–GBS pairs; GBS-vs-WGS scores fall below them.
Every export script must be pointed at a stage (ATLAS_DIR=<stage>). Several (o01, o17) write into the served tree when it is unset.
Lepton reads from AWS, not PBSS (aws_bridge.sh); never compute on a cluster login node.

Decisions for the PI (the records do not settle these; decided ones are marked)

#	Question	Where it arises
D1	~~When samples are added to an existing cohort, refit?~~ Decided (PI, 2026-10-08): a cohort is one experiment, delivered in aggregate. Samples are never added to an existing cohort; new samples (even of genotypes we already hold) form a new cohort, and every model (NA/EU EM, dosage) is fitted per cohort. Same-plant links to earlier cohorts go through `sample_identity` (rule 6).	all read types (S5)
D2	Should Havill, divpanel, USDA and Saaz move from route V to route B?	GBS §4 item 10; GRAS-Di G7
D3	~~Depth-ratio sex calls vs `ASSAYS.md` §3.1 / 2026-09-16?~~ Resolved 2026-10-08 (PI: valid with the right dosage correction): MSY-only depth, mother-masked, informative bins, k = 2–3 mixture (Y); PAR-summed, k-mer-backed or pan-X k-mer dosage (X); within a cross with parents. Addenda under both ASSAYS entries. Not validated for panels. Open: the served sex lane uses the 09-24 depth caller only; it disagrees with the 10-05 k-mer calls on 10 plants (5 Cascade, 4 Saaz, 1 Wye).	GBS §4 item 11
D4	Confirm rule 4 (one alignment per cohort for all layers).	GBS, GRAS-Di
D5	Mixed-assay cohorts (GBS progeny + WGS parents) share one EM parameter set. Keep?	GBS §1; WGS §5 item 11
D6	Apollo atlas rows use ~2× subsampled BAMs (inferred), not the ~9× ones. Intended?	WGS §5 item 3
D7	Divpanel multi-barcode runs (≥ 15 contain another plant) are in the atlas as deposited. Split them?	GBS §4 item 3

Record errors found during reconstruction (not yet fixed unless noted)

components/DOSAGE.md called hop_adosage unmerged; it was merged in 029bf81 (fixed 2026-10-08).
annevo_eval/sample_identity/sample_identity.tsv row kr:KR lists DRR024404–407, which ENA gives as SW (SAMD00021060, PRJDB3233), not KR's DRR024452–456 (SAMD00021059, "Humulus lupulus KR"; checked 2026-10-08). The reads aligned for the KR atlas row are KR's; only the identity table's run column is wrong (fix in e01_identity_table.py, then re-run; the table is never hand-edited).
Saaz "non-progeny" (backfill report rows 1–12): two different groups, each with its own origin.
DRR628685–690 are deposited cultivars, not progeny: titled SW, Agnus, Kazbek, R6KK1, R6M2, R6K4 in /data/hop_data/PRJDB19078_files.tsv; first labelled in sync/2026-09-04/lepton_zenith/PLAN_saaz.md.
DRR613305/306/307/388/435/436 are titled as progeny (S111–113, S194, S241–242) and fail two genotype tests against the parents (sync/2026-09-29/clade_ab_imputation/tables/T12b_inconsistent_runs_per_plant.tsv; sync/2026-09-30/phased_parents_switch_concordance/README.md).
(2026-10-08: unified from the manifest, SAAZ_EXCLUSIONS.md; deployed 2026-10-08 22:15, see CHANGELOG.) Exclusion lists differed by script (I-28): MC/UA exclude only the six DRR613xxx; 09_evaluate only the cultivars; s19/h00/sexscan all 12; phasing may include the cultivars.
annevo_eval/docs/INTROGRESSION_ATLAS_TECHNICAL.md describes the Wye RNA lane as route V; the served lane is the o13 BAM pileup. §10.2 says the OSU sync scripts folder is empty (it is not).
Catalog PRJNA694508_rnaseq_alignments says GPU pbrun; the 21110M BAMs that feed the atlas came from CPU STAR + samtools markdup.
CLAUDE.md and APOLLO_ASSEMBLY_PROCESS.md §9 still call the phased Apollo embargoed; the Carlsberg phased assemblies (GCA_060344695/715) are public and uncatalogued.
Key record folders are not in git: cluster/lepton_linear/ and several dated sync/ script folders.

Pipeline plan

Built (2026-10-08, increment 1): manifest loader/validator + jobs generator in pipeline/ (ingest.py validate|jobs|show; record pipeline/METHODS.md); manifests for 9 cohorts in hop_atlas/cohorts/ (8 back-filled + carey2026 written fresh; carey2026 also carries its numbered stage scripts pipeline/c26_*, which took it to the served atlas on 2026-10-09); back-test reproduces all 30 historical route B / lineage / sex job tables (30/30 match, 10/10 must-fail controls detected; /data/hop_data/hop_atlas/runs/ingest_backtest/). Next: call + export.

The fix is the one ADDING_DATA.md names: a cohort manifest read by one parameterised command per stage, replacing the copy-and-edit forks. The runbooks above say exactly which constants each fork carries, which is the manifest's schema.

Manifest (one directory per cohort, hop_atlas/cohorts/<cohort>/): - cohort.yaml: key, title, assay (gbs_apeki_se | gbs_apeki_pe | gbs_psti_apeki_pe | grasdi | wgs | rnaseq), design (cross | panel), parents, reference, truth plants, triploids, exclusions with reasons, atlas group/preset, limited (non-ApeKI-GBS lineage handling), pedigree line. - samples.tsv: sample key, run accessions, role (progeny/parent/panel/truth), assay and layout per library (mixed-assay cohorts), source + md5 + bytes + read count, label, verified sex + source, identity group, exclude reason.

Stages (each reads the manifest, writes /data/hop_data/cohorts/<cohort>/<stage>/ with .done markers, and is safe to re-run):

Command	Generalises	Notes
`fetch`	OSU `fetch_one.sh`, Carey `sra_stagein_worker.sh`/`fastq_worker.sh`, Wye ENA md5 script	md5 + read-count gates, PBSS copy
`qc`	`OG/scripts/02_enzyme_assay.py`, Carey `build_sample_table.py`, `paternity_check.py`	enzyme per run/mate, platform from headers, identity
`align`	writes a `cluster/lepton_linear` config + sheet from the manifest	dup rule from assay
`jobs`	— (no generator exists)	count job tables for route B, lineage, sex, dosage
`count`	`m01_count_one.sh`, `l02_count_one.sh`, `s10_pileup_one.sh`, `kt`	one runner, target list per layer
`call`	`o03`/`o04`/`o05`	TRIP/TRUTH/parents from the manifest; fit once per cohort (D1)
`export`	`o07`/`o08`/`o17` (+ `o15` for RNA)	applied to a stage, not to dated backups
`layers`	`c01`, `t03`, `o11`, `o01`; `atlas_layers.py`	replace hard-coded `COH`/`LIMITED_COH` with the manifest
`dosage`	09-25 manifest + the ~6 lists	`hop_adosage` package
`deploy`	`atlas_layers.py` / `atlas_deploy.py`	unchanged

Build order: manifest + loader and jobs first (no served change; unblocks route B for any cohort), then call + export (the base-atlas generator, ADDING_DATA.md gap 1), then layers, then dosage. Each stage is checked by re-producing an existing cohort's served output (OSU for GBS, Saaz for GRAS-Di, Apollo for WGS, Wye RNA for RNA-seq) before it is used on a new one.