Hop atlas home · Overview · Changelog · Architecture · Adding_Data · Introgression · Ribbon · Dosage · Phylogeny · Bundle · Ingest · Ingest_Gbs · Ingest_Wgs · Ingest_Grasdi · Ingest_Rnaseq · Ingest_Assembly

Rendered from hop_atlas/ingest/GBS.md (2026-10-13)

Ingesting GBS hop samples: SRA/ENA → hop atlas (:8900), reconstructed from project history

Status: runbook reconstructed 2026-10-08 from the dated records; not an executed pipeline. Every step cites the file it rests on. Where the record is incomplete, contradictory or where I inferred something, it says so: (inferred), GAP, CONFLICT. Companions: hop_atlas/ADDING_DATA.md (data type × atlas audit), hop_atlas/components/{INTROGRESSION,DOSAGE}.md, annevo_eval/docs/INTROGRESSION_ATLAS_TECHNICAL.md (code-level detail of the base atlas, snapshot 2026-10-03), annevo_eval/docs/ASSAYS.md (gotchas; "ASSAYS §x" below). Never: run compute on a cluster login node; mark duplicates on GBS; apply a GQ filter; rm -rf anything (CLAUDE.md).

Path abbreviations (same as the technical doc):

1. Scope: the GBS kinds we have handled

The records show three GBS library kinds. The "Cascade = 92 PstI + 88 ApeKI halves" reading is superseded: every Cascade progeny is a PstI+ApeKI double digest, one enzyme per mate; 92/88 is which mate carries which remnant (ASSAYS "2026-09-05 — Cascade GBS is a PstI+ApeKI DOUBLE DIGEST"; memory cascade-gbs-two-enzyme-split, correction 2026-08-28/09-05). No hop cohort is MspI (ASSAYS §1.5).

How the handling differs, in one line each: - Duplicates: never marked for any GBS library (all kinds). K2/K3 parents are WGS and are dup-marked; their pileups use --ff …,DUP (ASSAYS 2026-09-25 "--no-markdups is for GBS/GRAS-Di ONLY"). - Trimming: only Havill was trimmed before alignment (custom CWGC anchor + fastp). Every other cohort went to pbrun raw; barcodes/spacers are soft-clipped by the aligner (OG/README.md §4 table: 99.5 % of barcoded reads 5′-clipped ≈ 8 bp). - Layout: K1 → --in-se-fq(-list); K2/K3 → --in-fq(-list). Never use paired handling on SE (LL/config/usda_21110M.env comment). - Mixed-assay cohorts (K2, K3): progeny GBS + parents WGS share one EM parameter set in the NA/EU model (tech doc §4.2, issue I-20), parents get WGS rows in the count job tables, and pseudo-testcross marker definition from parents is impossible for Cascade (memory cascade-gbs-two-enzyme-split). - Tag-space layer: kt --mode gbs5scan reads an ApeKI tag at the read start, else scans the read for ApeKI sites — this is how Cascade's PstI mates enter (LR2/scripts/kt.cpp:1-7). PstI-anchored tags as such are not catalogued (inferred from kt.cpp).

Out of scope here (reduced-representation or genotype-only, not GBS reads): GRAS-Di Saaz×HL1012 (ASSAYS §1b; route V), DArT Pilgrim (sync/2026-10-07/pilgrim_in_atlases/; genotypes only), Wye leaf RNA-seq (OA/scripts/o13, o15), and the Zhang 2017 Hopsteiner HapMap (4,512 GBS samples as a TASSEL genotype matrix on arbitrary Shinshu Wase pseudo-chromosomes, no reads; lift-over project in progress, not in any atlas: sync/2026-10-08/zhang2017_hapmap_project/PROJECT_BRIEF.md, memory zhang2017-hapmap-and-supplement).

2. Stages

Legend for "Generic?": GENERIC = runnable for a new cohort by config/arguments; COPY-AND-EDIT = a cohort-specific script you copy and edit (what to edit is listed); RULE = written rule, no tool.

Stage 1 — Download, manifest, cold-tier copy

• Tools/flags: curl/wget, sra-tools 3.2.0 (OSU), archive.sh (data-archive skill), aws_bridge.sh to-aws for Lepton.
• Outputs: /data/hop_data/<cohort>/…fastq.gz + manifest; PBSS s3://general-purpose/skothenhill/archive/<cohort>_reads/ (e.g. wye_PRJNA1357051_reads/, cascade_PRJEB63534_reads/, PRJNA1219636/reads/, havill_clean_reads/, diversity_PRJEB63136_reads/).
• QC gates used: md5 vs ENA/NCBI (Wye, OSU); line count %4 (OSU); platform + read length decoded from read headers, not ENA labels (build_manifest.py PLAT table; memory sequencing-platform-labels-unreliable).
• Gotchas: gzip -t cannot detect truncation; gate on the archive's published size (ASSAYS §4.2j, "2026-09-05 — gzip -t passing…"; Cascade repair_dl.sh used gzip -t only). ENA fastq_bytes ≠ staged size (ASSAYS "ENA fastq_bytes is NOT the staged file size — 2026-09-08"). ENA .gz and submitter .bz2 are read-identical but header-different (ASSAYS 2026-09-05). Bulk SRA to Lepton: use ODP S3, not ENA FTP (ASSAYS "Bulk ENA/SRA download for Lepton … 2026-10-06"; "Akagi Omni-C … 2026-09-24"). Havill: 272 runs are 136 libraries sequenced twice — never delete a block (memory havill-cohort-two-sequencing-runs).
• Generic? COPY-AND-EDIT. Best templates: OSU fetch_one.sh (SRA route, strongest gates) and the Wye script (ENA route with md5). Edit: accession, output dir, run selection (select_runs.py has hard-coded panel names and library-name rules), group names.

Stage 2 — Read QC: enzyme, inline barcodes, demultiplexing, trimming, sample identity

Enzyme (always measure; ASSAYS §1.5): - annevo_eval/scripts/gbs_enzyme_survey.sh (2026-08-04): read-start 5-mer enrichment vs offset 30; reference motif at aligned 5′ ends (strand/CIGAR-aware). - OG/scripts/02_enzyme_assay.py (2026-10-02, most reusable): first 20,000 reads per run, best offset 0–12 for C[AT]GC (ApeKI), TGCAG (PstI), CGG (MspI), TGCAT (EcoT22I); modal barcode. Output OG/tables/enzyme_by_run.tsv. Known false "UNRESOLVED": barcode containing CTGC (horizon, OG/README.md §2). - K3 per-mate check: the awk one-liner in ASSAYS "2026-09-05 — Cascade GBS is a PstI+ApeKI DOUBLE DIGEST" (count TGCAG vs CAGC|CTGC at position 1 on each mate).

Barcodes / demultiplexing: sync_archive/2026-08-04/gbs_barcode_enzyme_audit/scripts/01_barcode_census.py, 02_split_run.sh, 05_concordance.py (divpanel census + barcode-group split + concordance). Rule: per-read trim at the first CAGC|CTGC within 32 bp, SEQ and QUAL at the same offset; drop adapter read-through (ASSAYS §1.6). No cohort in the atlas was re-demultiplexed: divpanel's multi-barcode runs enter as deposited (199 rows in route V, tech doc §1.2) (inferred: no split version is referenced by any atlas script).

Trimming actually applied:

Sample identity / label QC (cohort-specific, run before anything is trusted): - Havill: block-B swap audit sync/2026-08-26/scripts/30_havill_sample_assignment_audit.sh, 32_havill_full_matrix.py (read-level allele test, MAF-filtered sites 0.20–0.80; bcftools gtcheck fails here) → repair sync/2026-08-27/scripts/60_havill_split_blocks.sh (memory havill-block-b-sample-swaps). - Wye: duplicate accessions + non-progeny family (sync/2026-09-16/wye_anomalies/; ASSAYS "Pedigree / contamination assays … 2026-09-16"). - OSU: KING/IBS0 parentage OG/scripts/09_king_inputs_local.sh, 10_king.py, 14_parentage_king.py; BioSample sex unreliable → verified sex sync/2026-10-04/osu_sex_verification/tables/V1_osu_sex_verification.tsv. - Generic tool now available: annevo_eval/scripts/paternity_check/paternity_check.py (AD-only, calibrated nulls; tests for osu/havill/cascade in tests/). - Same-plant metric: opposite-homozygote rate, not genotype concordance (ASSAYS §1.6).

• Generic? RULE for enzyme/barcode/trim; scripts are COPY-AND-EDIT (hard-coded manifest paths and N). Identity QC: paternity_check.py is GENERIC; the rest is cohort-specific.

Stage 3 — Alignment

• Reference: REF (21110M male composite; the only hop reference with ChrY; Chr01–05 and ChrX reverse-complemented vs drHumLupu-forward; tech doc §1.1). Every atlas layer is in this frame.
• Tool: GPU pbrun fq2bam (clara-parabricks 4.7.0-1; 4.7.1-1 on EOS/Saaz). --no-markdups for every GBS library. Read group must carry ID, SM, PL, PU, LB (ASSAYS §4.2). Pooling many samples per call: constant SM:POOL, sample in ID/PU/LB, samtools split -f '%!.bam' by RG ID, then samtools reheader SM (ASSAYS §4.2b, both footguns).
• Which alignment feeds which atlas product (several cohorts were aligned more than once; the atlas layers do not all read the same BAMs):

• QC gates: samtools view -c -f 1024 must be 0 for GBS, >0 for dup-marked WGS (LL/README.md "Five things" #2); primary BAM records == ENA/NCBI read count when no trimming (READCOUNT_MANIFEST, USDA; OG/README.md §3: 371/371, 391/391); distinct @RG SM == number of BAMs (scripts 26/28 FATAL on collision); classic bwa index present with .ann length == .fai sum (LL/upload_refs.sh; ASSAYS §4.2f).
• Gotchas: dup trap (ASSAYS §1.1; PE GBS still 38.6 % flagged); old Cascade-parent BAMs carry no DUP flags (ASSAYS 2026-09-25); pooled markdup needs a unique LB per sample (ASSAYS 2026-09-25); --low-memory local only (§4.2c); pbrun illegal-memory-access on untrimmed long/odd WGS reads → fastp first (ASSAYS 2026-09-25, 2026-09-30); symlinked --ref breaks index lookup (ASSAYS "Cascade contig-junction validation … 2026-10-05"); old Cascade/Havill BAMs with default SM:sample merge into one sample (script 26 header); Havill first 128 progeny BAMs from delivered FASTQs contain 20 swaps unless split/reassigned (memory havill-fastq-must-be-cleaned-before-use); Lepton must not pull from PBSS (§4.2e; use AWS tier, annevo_eval/data_inventory/aws_bridge.sh).
• Generic? GENERIC: LL/linear_job.sh + LL/config/<pop>_<ref>.env + LL/sheets/<pop>_sample_sheet.tsv + LL/upload_inputs.sh <cfg> + LL/submit_linear.py <cfg> (README "Run it"). For a new GBS cohort write a config with PAIRED, MARKDUP=0, CALL_FF=UNMAP,SECONDARY,QCFAIL, GBS_BATCH, DO_FASTP, PARENT_MARKDUP (1 only if parents are WGS), NO_PARENTS/UNTARGETED for panels, READS_FS1 or READS_URI, PUSH_SUBS (keep bam if later layers must read the BAMs from PBSS — see GAP 9). MR/scripts/lepton/job_ref.sh (multi-reference, pooled, per-assay dup rule) is the other generic-ish aligner but its manifest (MR/scripts/00d_build_manifest.py) hard-codes the five crosses.

Stage 4 — Variant calling / allele counting at the diagnostic catalogue

Two routes produce per-site A/B read counts at the clade catalogue CIW/diag/targets_all.tsv.gz (29,218,128 sites; tech doc §2).

Route V (joint VCF → AD), used for the served Havill, divpanel, USDA calls: - Joint call (same flags in scripts 26/28, sync/2026-08-27/scripts/64_havill_recall_3cells.sh, LL mq20 pass): bcftools mpileup --min-MQ 20 --min-BQ 20 --ff UNMAP,SECONDARY,QCFAIL -a FORMAT/AD,FORMAT/DP | bcftools call -m -v --format-fields GQ | bcftools view -m2 -M2 -v snps, per chromosome (local) or 5 Mb windows (Lepton, WINDOW_MB=5, MAX_DEPTH 250/500). - Extract: python3 CI/scripts/07_extract_cohort.py <cohort> <vcf|presigned-url##idx##csi> [regions|-] [diag_dir] → CIW/cohorts/<cohort>.npz (keeps {REF,ALT}=={a,b}, one site per 150 bp grid cell; no QUAL/DP/GQ/GT filter). - Gotcha: variants-only VCFs have no reference-allele evidence → depleted bins (ASSAYS "2026-09-30 — Variants-only call sets hide reference-allele evidence"; memory variants-only-site-selection-trap). This is why Cascade/Wye moved to route B. Havill/divpanel/USDA are still served from route V.

Route B (BAM pileup at catalogue sites), used for the served Cascade, Wye, OSU calls: - MC/scripts/m01_count_one.sh <cohort> <sample> <pbss_key> <GBS|WGS> [targets]: BCF mpileup -f REF -T targets --min-MQ 20 --min-BQ 20 --ff UNMAP,SECONDARY,QCFAIL[,DUP if WGS] -a FORMAT/AD -Ou <presigned url> | bcftools query -f '%CHROM\t%POS\t%REF\t%ALT[\t%AD]\n' → MCW/bamcounts/<cohort>/<sample>.tsv.gz + .done; FAIL if ≤ 1,000 lines. Driver MC/scripts/m01_run_jobs.sh <jobs.tsv> <P> (xargs -P; WGS rows first). Job table MCW/jobs/cw_jobs.tsv (+ cw_rerun*.tsv) = cohort sample pbss_key GBS|WGS. - OSU on Lepton (BAMs on fs1): OA/scripts/lepton/submit_count_osu.py --workers 6 --par 4 → count_osu_aims.sh (same command; md5-checks shipped bcftools/fai/targets; drops the ≤1,000-line rule → 9 near-empty libraries kept). Equality check: OA/local_test/run_one_local.sh 0227-036 (byte-identical). Return: Lepton → AWS → aws_bridge.sh to-pbss → aws s3 sync to OA/bamcounts/. - Matrix: MC/scripts/m03_build_bam_cohort.py <cohort> (OSU copy OA/scripts/o03_build_bam_cohort.py, sample list from env SAMPLES) → MCW/cohorts/<cohort>_bam.npz; keeps sites read in ≥ 5 samples (MINS), one per 150 bp grid cell (GRID). - Gotchas: mpileup -T emits indel+SNP at one POS → dedupe (ASSAYS §1.7); implicit -d 250, BAQ on, anomalous pairs skipped (tech doc issue I-21; K2/K3 are PE); supplementary alignments are admitted by --ff UNMAP,SECONDARY,QCFAIL (ASSAYS 2026-09-17); OSU catalogue bamcounts read ~0 Target-B reads at lineage marker sites — do not use them for haplotype-specific alleles (ASSAYS "DArTseq 0/1/2 codes … OSU atlas bamcounts fail at Target-B markers (2026-10-05)"); never edit a shell script while xargs workers run it (ASSAYS 2026-09-24, 2026-09-30). - Generic? m01_count_one.sh is nearly GENERIC (arguments) but hard-codes REF, BCF, output root MCW/bamcounts and PBSS bucket; GAP: no script generates the job table (tech doc §1.2) — write cohort sample key GBS|WGS rows by hand. m03 COPY-AND-EDIT (input dirs); o03 is the better template (env SAMPLES). 07_extract_cohort.py GENERIC (arguments) but writes into the 09-29 product dir.

Stage 5 — NA/EU (clade A/B) base calls and base viewer files

• Fit + call: MC/scripts/m04_impute_multires.py <callset> (OSU copy OA/scripts/o04_impute_multires.py). A callset ending _bam reads MCW/cohorts/; others read CIW/cohorts/ (tech doc §1.2). EM once at 5 Mb (start e 0.02, dA 0.1, dB 0.2, w 0.8, η 0.02; exactly 10 iterations), held fixed at 0.5/1/2/5 Mb; τ = 0.5 from CI/tables/T2c_tau.json; call if pmax ≥ 0.95 and ≥ 3 sites; per-site cap nA+nB ≤ 16; independent bins (no HMM). Route V cohorts first run CI/scripts/08_impute.py <cohort>. Outputs MCW/calls/<callset>_<res>_calls.tsv.gz, MCW/imputation/<callset>_params.json.
• Edit per cohort: TRIP (triploids; m04/o04 lines ~46-58), TRUTH (truth plants; leave-plant-out sites), CROSS_VCF; OSU added TRUTH["osu"]={"21110m":"M21"} and a name rule for known triploid cultivars.
• Evaluate: MC/scripts/m05_evaluate.py (served callset per cohort hard-coded at m05:22-26 → MC/tables/Q0_callsets_used.json; must run before m06/m07/m11), m06_transitions.py, m06b_flip_hotspots.py, m11_master_table.py; OSU: OA/scripts/o05_evaluate_osu.py. Gate = viewer warning rule: callable < 0.90 OR truth accuracy < 0.99 OR isolated parent-inconsistent > 0.005 (tech doc §4.5).
• Labels: OA/scripts/o00_labels.py → OA/labels/<cohort>_sample_labels.tsv, atlas_alias_additions.tsv (labels kept separate from calls; parents hard-coded in PARENT). Name maps for panels: /data/hop_data/divpanel/run2sample.tsv, /data/hop_data/usda_21110M/usda_run2cultivar.tsv.
• Export into a stage (the add-a-cohort template is OSU): ATLAS_DIR=<stage> python3 OA/scripts/o07_viewer_export.py (hard-coded COH='osu', CS='osu_bam', MO,FA='cascade','19058m', expected count 826, string patches of PANEL_ORDER and the single-sample parent map in index.html; restarts from OA/viewer_backup_pre_osu/) → o08_preset_patch.py (re-run after every o07) → o17_detail_osu.py <cohort> (detail files written directly into the served tree /data/hop_data/universal_introgression_atlas/viewer_detail_mr/). The original seven cohorts were exported together by MC/scripts/m07_viewer_export.py + m07b_check_viewer.py and UA u14/u16/u17 (tech doc §10.1).
• Tests: OA/scripts/viewer_tests/run_tests.sh <res> × 4 resolutions (p30 p15 p16 p11 p12 p07 k21 + cohort test o21_osu_rows_test.py pattern: table truth, shifted must-fail, pixel equality, 404 control) → p08_test_summary.py.
• Gotchas: EM is fitted per callset on all samples including parents and truth plants; GBS progeny and WGS parents share dA/dB (I-20); fitting is not documented for adding samples to an existing cohort (GAP 7); o07/o15/o08 each restart from their own backups (I-1) — run order matters (tech doc App. C2(e)).
• Generic? COPY-AND-EDIT throughout (ADDING_DATA.md gap 1: "no generator for the base introgression atlas"). Edit: cohort key, callset name, parents, truth plants, triploids, expected counts, panel groups, preset, sample list (OA/osu_samples.txt is hand-made, tech doc §10.2 step 1).

Stage 6 — Layers

6a. Subspecies / lineage layers (components/INTROGRESSION.md §4.3; LR1/PIPELINE.sh, LR2/PIPELINE.sh)

1. SNP layer counts: LR1/scripts/l02_count_one.sh <bam> <sample> <outdir> <GBS|WGS> at LRW/catalogue/targets.tsv.gz (GBS keeps DUP; WGS uses targets_wgs500.tsv.gz + DUP). Drivers: l02_run_divpanel.sh, l02_run_list.sh <cohort> <jobs.tsv> <P> (Havill from havill_blocks/{A,B}, Cascade/Wye from PBSS m21/m21_wyefix). Lepton for fs1 BAMs (USDA, OSU): LR1/scripts/lepton/{make_tasks.py,submit_count_l02.py,count_l02.sh,pull_counts.sh} (PIPELINE_lepton.sh). Then l03_run_all.sh <cohorts> / l03_build_sparse.py (drops indel/SNP collisions, ASSAYS §1.7).
2. Calls v6: LR2/scripts/c01_run_all_v6.sh → c01_call_levels_v6.py <cohort> --groups sync/2026-10-06/lineage_refinement/tables/C00_groups.tsv --tag _v6 → LRW/calls/<cohort>_v6/.
3. Tag space: samtools view -F 0x900 <bam> | kt --mode gbs5scan --out <s>.kt (LR2/scripts/kt_jobs.py, kt_run_gbs_local.sh; Lepton LR2/scripts/lepton/{submit_kt.py,kt_worker.sh,pull_kt.sh} for USDA/OSU) → t03_run.sh <cohort> → EXTRA='--minref 1' c01t_run.sh _t4 <groups> <cohort> with --l0-from LRW/calls/{coh}_v6 (as in c01_run_all_v6.sh) → LRW/tagspace/calls/<cohort>_t4/.
4. Register the cohort: add to hard-coded COH in LR2/scripts/e03_viewer_export.py:28, e04_detail_export.py:18, LR2/viewer/l23_lineage_v3_test.py:25; LIMITED_COH / LIMITED_KEYS for non-ApeKI-GBS libraries (WGS parents of K2/K3 are already in LIM_KEYS).
5. Export + test: python3 hop_atlas/tools/atlas_layers.py lineage $STG then test.
6. Gotchas: group allele frequencies need reference data for every competing group (ASSAYS "Group allele-frequency models …"); ApeKI-GBS-trained lineages do not transfer to other assays (hence LIMITED_COH); exact allele-centred k-mers can lose a haplotype (ASSAYS 2026-09-24).
7. Generic? COPY-AND-EDIT (cohort lists hard-coded in c01_run_all_v6.sh, kt_run_gbs_local.sh, e03, e04, l23; job lists built by kt_jobs.py/make_tasks.py with hard-coded cohorts (inferred from their use in PIPELINE.sh; not read line-by-line)). Export/test/deploy GENERIC (atlas_layers.py).

6b. Sex lane (ChrX sex-aware, ChrY lineage) (SX/PIPELINE.sh; tech doc §8)

1. Pileup: SX/scripts/s10_pileup_one.sh <cohort> <sample> <pbss key|local bam> <GBS|WGS> at SXW/catalog/targets_sex.tsv, -r ChrY,ChrX:225000001-268239420, same MQ/BQ/--ff rule as m01; driver s10_run_jobs.sh SXW/jobs/s10_jobs.tsv 8. VCF-only cohorts: s12_usda_vcf.sh (FORMAT/AD; variants-only limitation). OSU: Lepton OA/scripts/lepton/count_osu_sex.sh via submit_count_osu.py --body count_osu_sex.sh.
2. s13_build_counts.py → s14_sex_table.py (sex source: crosses = the 09-24 MSY read-ratio caller sync/2026-09-24/per_chromosome_dosage/tables/combined_sample_x_chromosome.tsv; panels = name rules; Havill phenotype as independent label) → s15_y_lineage.py → s16_x_caller.py --no-mask → s16b_hxr_male_mask.py → s16_x_caller.py → s24_viewer_export.py → s27_viewer_integrate.py.
3. New cohort precedent (OSU): ATLAS_DIR=<stage> python3 OA/scripts/o11_sexchrom_osu.py merges the cohort into the existing sexchrom/atlas_sexchrom.json (reference females = BioSample females; no HXR mask; 5 Mb only). Sex was later re-verified by X dosage + Y k-mers (sync/2026-10-04/osu_sex_verification/).
4. Then python3 hop_atlas/tools/atlas_layers.py xconc $STG (LR2/scripts/x11_sexaware_x_concordance.py; skips crosses without progeny/sex-aware parents).
5. Gotchas: Y coverage does not call sex from single-end GBS (ASSAYS §3.1); X:A and Y:A depth do not call sex on 21110M (ASSAYS "X:autosome and Y:autosome read depth do NOT call sex on the 21110M male composite"); Havill MSY depth has two XY-like levels — use a k=2–3 mixture (ASSAYS 2026-09-24); PAR exists twice in male_ref — sum Y-PAR onto X (ASSAYS "OSU sex-chromosome dosage … 2026-10-05"); X-clade choice changes X dosage (ASSAYS "European-X fix 2026-10-06"); ChrX/ChrY testcross markers on 21110M are artefact-dominated (ASSAYS 2026-09-05); BioSample sex unreliable (memory osu-biosample-sex-vs-y-mismatch).
6. Generic? COPY-AND-EDIT. s10 is argument-driven; s24 asserts exactly 1,272 samples and cannot regenerate the served file (tech doc issue I-34); the only add-a-cohort path is an o11-style merge.

6c. Phasing, crossovers, a/b/c concordance, transmission (crosses only; PP/PIPELINE.sh; tech doc §6)

• PP/scripts/p01_phase_concordance.py <res> (crosses havill, cascade_bam, saaz, wye_bam, apollo_bam hard-coded at p01:40, parents at p01:41-46) / OSU copy ATLAS_DIR=<stage> python3 OA/scripts/o01_phase_osu.py <res> × {5Mb,2Mb,1Mb,500kb}; p03_bfreq_transmission.py (OSU: inline in o07); validation p05_validate_concordance.py; OSU vs KING o10_concordance_vs_king.py; summary o06_transmission_summary.py.
• Inputs: callset calls, MCW/stage1/hap_bin_<res>.tsv, multires/atlas_samples_<res>.json, exclusions from multires/atlas_transmission_5Mb.json[cross].excluded.
• Gates: positive controls (Cascade triploids → class c; Saaz non-progeny → c), random-flip control (tech doc §6.2–6.3).
• Gotchas: o01 writes straight into the served atlas if ATLAS_DIR is unset (tech doc §6.6); ChrX phased as an ordinary diploid (I-29); panels (divpanel, USDA) have no phasing; default exclusions are hand-chosen per cross (tech doc §6.4; Havill mixed paternity is not reflected, I-28).
• Generic? COPY-AND-EDIT (cross, parents, exclusions); then atlas_layers.py xconc GENERIC.

6d. Pedigree

• One CROSS line (cohort → mother, father, label) in LR2/scripts/p01_pedigree_build.py; name synonyms in SYN/APPROX/EXCL; then atlas_layers.py pedigree $STG; review tables/P01_pedigree_matches.tsv, P01_unmatched_nodes.tsv (components/INTROGRESSION.md §3). GENERIC after the one-line edit.

Stage 7 — Dosage atlas (components/DOSAGE.md)

• Counts (GBS rule): 09-24 sync/2026-09-24/per_chromosome_dosage/scripts/01b_count_bins_1k.sh / 02b_run_manifest_1k.sh (samtools view -q 20 -F 0xB04, DUP kept, 1 kb read-start bins; manifests built by hand, PIPELINE.sh step 0). 09-25 MR/scripts/lepton/job_ref.sh per reference (submit_pcd.py <ref> --bundle …): pooled 96 GBS per pbrun, --no-markdups; count_one.sh / count_one_v2.sh (1 kb GBS bins), joint call mpileup -q 20 -Q 20 --ff UNMAP,SECONDARY,QCFAIL -d 500 | call -m -v, ab_hist.py. Manifest MR/scripts/00d_build_manifest.py (895 samples; hard-coded cohort read sources).
• Allele balance: 03b_run_ab.sh <pop> / 03_allele_balance.py (AD fraction at DP ≥ 8 / ≥ 20; parents typed by own AD at DP ≥ 10; never GT).
• Build/deploy: /data/hop_data/dosage_explorer_v2/stage/scripts/rebuild_and_validate.sh (build_all_cn.sh → worked_example.py → fill_defs.py → make_page.py → validate_cn.py), hop_atlas/tools/relocate_dosage.py, atlas_deploy.py {stage,build,test,deploy} dosage. Hard-coded lists to edit: build_all_cn.sh, validate_cn.py, check_cn_browser.py (REFS,POPS), v1 build_all.py, mr_config.py (REFS), build_cn.py (CALLSRC,FRAME,REFHAP,EXCLUDE_BASE) (DOSAGE.md §2).
• Coverage of GBS cohorts: Havill, Cascade, Wye served; divpanel/USDA only in the 09-24 tables (not served); OSU only as the sex panel (sexpanel/, source sync/2026-10-05/dosage_explorer_v2_sexpanel → archived in /data/hop_data/hop_atlas/dosage_sources/).
• Gotchas: GBS depth only in informative 1 kb cut-site bins (ASSAYS 2026-09-24); never GQ (ASSAYS §2.1); per-plant depth cap on read stacks (§2.9) and over callable sites (ASSAYS 2026-09-30 depth cap); per-chromosome GBS depth is not a trisomy test without kNN normalisation (ASSAYS "GBS paternity / dosage on the OSU panel … 2026-10-04"); tier-matched normaliser for Y/X k-mers (ASSAYS 2026-10-06); ploidy calls only within a reference-matched background (§2.5c).
• Generic? COPY-AND-EDIT (DOSAGE.md §3: "no tested procedure"). Note: the generic method packages are now merged into annevo_eval/methods/{autosomal_dosage (hop_adosage),sexchrom_dosage} (commit 029bf81); DOSAGE.md still calls them unmerged (stale doc), and the served atlas is not built from them.

Stage 8 — Stage, test, deploy, register

• Introgression (current tool): python3 hop_atlas/tools/atlas_layers.py stage --name <n> → edits on $STG → serve $STG --port 8907 → test $STG → deploy $STG (refuses if :8900 changed since stage; backs up *.bak_<date>_<name>; deletes nothing) → test on :8900 (components/INTROGRESSION.md §2). atlas_layers.py patch $STG re-applies the 32 lineage + 4 pedigree hooks if a cohort script rewrote index.html (o07 does).
• Historical base-atlas swap: OA/scripts/o09_swap_into_served.sh (md5-guarded vs backup/manifest; OA/served_after_swap.md5) — superseded in practice by atlas_layers.py deploy (inferred: both exist; the docs now direct to atlas_layers).
• Dosage: atlas_deploy.py stage|build|test|deploy dosage + check_dosage_http.py.
• Approval and record: PI approval for any served change; entry in hop_atlas/CHANGELOG.md; annevo_eval/data_inventory/register_product.sh <path> "<why>" [--feeds …] for counts/labels/calls; catalogue entry in catalog.yaml then refresh.sh (tech doc §10.2 step 14); PIPELINE.sh in the dated sync/ folder (CLAUDE.md analysis protocol; no raw data in sync/).
• Generic? GENERIC.

3. Per-cohort differences (which variant of each stage)

4. Missing, ambiguous or contradictory steps

1. No generic downloader; USDA download undocumented. No script for PRJNA1219636 reads was found; divpanel and Havill downloads have no checksum step; Cascade repair relies on gzip -t, which ASSAYS §4.2j says cannot detect truncation.
2. No single enzyme/barcode QC tool. Three one-off implementations (gbs_enzyme_survey.sh, 01_barcode_census.py, OG/scripts/02_enzyme_assay.py). The 32 bp barcode search window rule (ASSAYS §1.6) vs Havill's 16 bp anchor window (havill_merge_trim.sh) — Havill carries spacers, not barcodes, so this may be intended; not stated.
3. Divpanel chimeric runs are in the atlas as deposited. ASSAYS §1.6 says ≥15 multi-barcode runs hold a different plant; no split version feeds any atlas script (inferred). Anomalous run ERR11580424 (no ApeKI remnant) is excluded from the 09-24 dosage reference but route V uses all 199 samples (inferred from the 199 count, tech doc §1.2).
4. Trimming policy is not uniform and partly undocumented. Havill trimmed; everyone else raw "for comparability". Wye progeny were fastp-trimmed in the 09-05 run but raw in the 09-25 run that feeds the atlas (inferred from code).
5. One cohort, several alignments. Havill's base/lineage layers read the script-26→60 BAMs (delivered FASTQs, run-level reassignment) while sex/dosage read the 09-25 fastq_clean BAMs (126; Z_034/035/041/057 absent). Cascade and Wye route V baselines (not served) come from older alignments than the served route B. A new cohort should use ONE alignment for all layers; the record never states this as a rule.
6. Havill residual defects. A third bad father run SRR22450427 is still in the father's reads (memory havill-fastq-must-be-cleaned-before-use, 2026-09-03, "rebuild decision pending PI"); recall3 excludes only SRR22450430 and SRR22452695 (60_havill_split_blocks.sh). About 15 further block-B files look multi-plant (memory havill-block-b-sample-swaps, update 2026-09-30). No atlas exclusion reflects these.
7. Adding samples to an existing cohort: no procedure. The EM is fitted once per callset at 5 Mb; nothing says whether new samples require a refit (ADDING_DATA.md; MC/README.md).
8. Route B job tables have no generator (MCW/jobs/cw_jobs.tsv, targets_apollo_500bp.tsv.gz; tech doc §1.2). OSU sample list OA/osu_samples.txt is hand-made.
9. USDA BAMs exist only on Lepton fs1 (PUSH_SUBS="status logs qc calls state"); any re-count (lineage, route B instead of route V) depends on fs1 still holding them (inferred risk). OSU BAMs do have a PBSS copy.
10. Route V still serves Havill, divpanel and USDA although the variants-only trap (ASSAYS 2026-09-30) was fixed for crosses by route B; the record does not say whether these three should move to route B. (The depleted-bin rule is applied for Havill as a VCF cross cohort, tech doc §3.3.)
11. CONFLICT — sex calling on GBS. Crosses take sex from the 09-24 MSY read-ratio caller (SX/scripts/s14_sex_table.py), while ASSAYS §3.1 ("Y-coverage does NOT call sex from single-end GBS") and ASSAYS 2026-09-16 ("X:autosome and Y:autosome read depth do NOT call sex on the 21110M male composite") reject depth-based sex calls. The records do not reconcile this; OSU was later re-verified by X dosage + Y k-mers. Not interpreted here. Resolved 2026-10-08: see ingest/README.md D3 and the addenda under both ASSAYS entries.
12. Sex JSON cannot be regenerated: s24_viewer_export.py asserts 1,272 samples (I-34); OSU was merged by o11. A new cohort needs another o11-style merge.
13. OSU record drift: OA/scripts/PIPELINE.sh omitted o01, o06, o08–o11, o15, o16 and the sex Lepton job (tech doc §10.2 step 15); the sync PIPELINE.sh now lists them. The sync scripts mirror is no longer empty (stale tech-doc statement) but differs from /data for o11, run_tests.sh, xytruth.py, o21.
14. Wye cohort hygiene not reflected in the atlas: 7 duplicate-accession pairs and 7 non-progeny "one family" plants (memory hop-wye-target-bl21-cross); the served Wye transmission exclusion list is empty (tech doc §6.4).
15. PstI tags in tag space: kt catalogues ApeKI-anchored 64-mers only; PstI mates contribute through the ApeKI-site scan fallback (inferred from kt.cpp header). Whether that loses Cascade information is not measured in the record.
16. Stale docs: components/DOSAGE.md says hop_adosage is unmerged (merged in 029bf81); tech doc §10.2 says the OSU sync scripts folder is empty.
17. Zhang 2017 HapMap: GBS genotypes with no reads and no atlas route yet (lift-over in progress, sync/2026-10-08/zhang2017_hapmap_project/).

5. Ordered checklist for a new GBS cohort (composed from the records above; never executed end to end)

1. Manifest + download (template: OSU fetch_one.sh or Wye download_PRJNA1357051.sh); md5 and published-size gates; decode platform/read length from headers; archive to PBSS; stage to AWS for Lepton.
2. Enzyme per run (OG/scripts/02_enzyme_assay.py; per mate for PE); barcode census if remnants are offset (ASSAYS §1.6); decide trim (default: raw, as USDA/OSU/divpanel); identity/label QC (paternity_check.py, opposite-hom rate).
3. Align with LL (new config/<pop>_21110M.env + sheet; MARKDUP=0, CALL_FF=UNMAP,SECONDARY,QCFAIL, WGS parents PARENT_MARKDUP=1, keep BAMs on PBSS). Gates: DUP = 0, reads = published count, unique SM.
4. Route B counts: hand-write cw_jobs.tsv-style rows → m01_run_jobs.sh (or the Lepton count_osu_aims.sh pattern) → o03-style matrix.
5. Copy o04 → set TRIP/TRUTH → calls at 4 resolutions; copy o05 → reliability; check the u16 warning rule.
6. Copy o00, o07, o08, o17 → export into an atlas_layers.py stage; run run_tests.sh × 4 + a cohort test (o21 pattern).
7. Crosses: copy o01 (with ATLAS_DIR set) × 4; o10/p05 checks.
8. Sex: s10 pileups → o11-style merge → atlas_layers.py xconc.
9. Lineage: l02 → l03 → c01_call_levels_v6.py → kt --mode gbs5scan → t03 → c01t _t4; add to COH in e03/e04/l23 → atlas_layers.py lineage.
10. Pedigree CROSS line → atlas_layers.py pedigree.
11. atlas_layers.py test → PI approval → deploy → test on :8900; CHANGELOG; register_product.sh; catalog.yaml + refresh.sh; dated sync/ PIPELINE.sh.
12. Dosage (optional): add to the 09-25 manifest and the six hard-coded lists; rebuild via atlas_deploy.py … dosage.