Hop atlas home · Overview · Changelog · Architecture · Adding_Data · Introgression · Ribbon · Dosage · Phylogeny · Bundle · Ingest · Ingest_Gbs · Ingest_Wgs · Ingest_Grasdi · Ingest_Rnaseq · Ingest_Assembly

Rendered from hop_atlas/ingest/WGS.md (2026-10-13)

Ingesting short-read WGS into the hop atlas web app (:8900): runbook reconstructed from history

Written: 2026-10-08. Status: reconstruction of what was actually run. No command below is new; every one is quoted from, or points at, a dated record. Nothing here is a tested end-to-end pipeline (see ../ADDING_DATA.md). Labels used: - GENERIC: parameterised, reusable as is (config/env/CLI argument). - COPY-AND-EDIT: a dated, cohort-specific script; copy it to your own dated folder and edit the named constants. - RECORD: documentation only ("not meant to be re-executed blindly"); read it, then write your own job. - (inferred): my reading of the records, not stated in them.

Path shorthands (same as annevo_eval/docs/INTROGRESSION_ATLAS_TECHNICAL.md, "TECH" below): MC = sync/2026-09-30/multires_clade, MCW = /data/hop_data/multires_clade, CI = sync/2026-09-29/clade_ab_imputation, CIW = /data/hop_data/clade_ab_imputation, PCD = sync/2026-09-25/per_chromosome_dosage_multiref, P = s3://general-purpose/skothenhill/archive/per_chromosome_dosage_multiref (PBSS), LR1 = sync/2026-10-06/lineage_refinement, LR2 = sync/2026-10-07/lineage_refinement_v2, L = /data/hop_data/lineage_refinement, SX = sync/2026-09-30/sex_chromosome_lineages, PP = sync/2026-09-30/phased_parents_switch_concordance, KR = sync/2026-09-30/cordifolius_KR, CAR = sync/2026-10-08/carey2026_hop_wgs, LL = cluster/lepton_linear (repo root, not annevo_eval/cluster/). REF = /data/hop_data/ref_21110M_male/male_ref.fa (21110M, Akagi numbering).

Untracked in git (2026-10-08 check): LL/, MC/, KR/, PCD/, CAR/. Only LR2/PIPELINE.sh and hop_atlas/tools/ are tracked.


1. Scope: WGS data that has entered (or tried to enter) the atlases

Cohort / samples Accessions Labelled vs true platform Depth Used for Atlas status
Apollo × Pub_740M (apollo, 186 rows) PRJEB63565. Mother ERR11639885 → Apollo_mother; father ERR11639886–889 (4 runs) → Pub_740M; progeny ERR11641697… → AxP_234_### (sheet LL/sheets/apollo_sample_sheet.tsv) Mother HiSeq 2500 (D00553R), reads 35–265 bp; father 4 lanes of one NovaSeq; progeny headers @A00275 (NovaSeq) (LL/config/apollo_21110M.env; ASSAYS 3371). RNA-seq/Hi-C of the same project are mislabelled (memory sequencing-platform-labels-unreliable) progeny ~9×, mother ~30×, father ~90× (memory apollo-cross-coverage-design). Atlas BAMs are a ~2× subsample (below) base NA/EU (route B apollo_bam), lineage/subspecies (LIMITED), sex lanes (X/Y, WGS 1 Mb + yb), phasing (Apollo h1/h2 assembly phase), dosage (all 6 refs) served
cordifolius KR (cordifolius, 1 row) PRJDB3233, DRR024452–456 GA-IIx (HWUSI-EAS1725), 2×75 bp, Phred+33 (KR/PIPELINE.sh) ~8× (memory kr-multiref-competitive-mapping) base NA/EU (route K, borrowed params), lineage (LIMITED), sex lane (KR = F, PI decision). Not in dosage atlas served
SW75 Shinshu Wase (control for KR) DRR024404–407 GA-IIx, 2×109 bp, Phred+64 (ASSAYS 8315) — KR comparator; lineage counts (kr_wgs jobs) not a served row (inferred)
Cascade × LB_male parents (WGS rows inside GBS cohort cascade) PRJEB63534: ERR11608298 (Cascade ♀), ERR11608479 (LB_male ♂; is the father, memory cascade-cross-father-is-present) 150 bp PE (ASSAYS 8261); instrument not decoded in any record found 259 M / 211 M reads; OSU deep QC: autosomes ~12×, ChrX 25× for ERR11608298 (sync/2026-10-02/osu_21110M_genotyping/README.md:99) parents of cascade_bam, sex lanes, phasing, dosage; also used as the OSU cross "deep mother" served
Wye Target × BL2/1 parents (WGS rows inside wye) PRJNA1357051: SRR35941433 → WyeTarget, SRR35941434 → BL2_1 labelled NovaSeq 6000, 250 bp PE (LL/config/wye_21110M.env) 8.3× / 9.0× raw; measured 3.5–4.7× at MQ≥20 (DATA_CATALOG Wye entry) parents of wye_bam, sex lanes, phasing, dosage served
Carey 2026 (carey2026) PRJNA1437418, 22 runs (/data/hop_data/carey2026_PRJNA1437418/carey2026_sample_sheet.tsv) decoded per run from headers: NovaSeq 6000 / NovaSeq X Plus (CAR/tables/carey2026_sample_qc.tsv) 6.8–10.8× autosomal on 21110M (7 runs done) lineage/subspecies calls computed; no base-atlas rows; not deployed in progress: QC table (2026-10-08 19:07) lists 7 SAMPLE_OK, 15 FAILED/absent
OSU deep samples (not atlas rows) Cascade WGS ERR11608298 re-aligned; Hi-C SRR37256446/7 treated as deep — — OSU joint GBS+deep call set, KING identity (sync/2026-10-02/osu_21110M_genotyping/PIPELINE.sh §9) not an atlas route

No WGS exists in our records for: Havill parents (GBS, 2 libraries each), divpanel / USDA / OSU panels (GBS), Saaz × HL1012 (GRAS-Di). The "comparative WGS" set (PMR-480/MN-101 etc.) was deleted 2026-10-02 (DATA_CATALOG comparative_wgs).


2. Ordered stages

S0. Decide the cohort shape first

Question Why it matters Where it was decided
Cross (parents + progeny) or panel (no parents)? phasing, transmission, NO_PARENTS=1, truth plants LL/linear_job.sh header; TECH §1.3
All WGS, or WGS parents inside a GBS cohort? dup rule per sample; EM parameters are shared per call set across assays (TECH §4.2, issue I-20) MC/PIPELINE.sh
Single plant? route K (borrowed parameters) KR/
Low-pass (≤ ~10×) progeny? calling must be joint + untargeted, no depth floors (memory apollo-cross-coverage-design) LL/config/apollo_21110M_untargeted.env

S1. Download (ENA/SRA → PBSS = system of record)

Records (all RECORD / COPY-AND-EDIT; no generic downloader exists):

Cohort Script Source / tool Integrity check used
Apollo annevo_eval/data_inventory/provenance_scripts/PRJEB63565/ncbi_worker.sh (+ watchdog6.sh) NCBI prefetch --max-size 200g → fasterq-dump --split-files --threads 4 → pigz (EBI throttled to ~0.85 MB/s) none adequate: gzip -t only → 10 truncated runs (below)
Apollo repair sync/2026-09-05/lepton_linear/APOLLO_TRUNCATED_RUNS.md submitter .bz2 → .gz with ENA headers rebuilt; PBSS PRJEB63565/reads_repaired/ bytes/md5 vs ENA submitted_*; records == ENA read_count/2; header form @<ACC>.
Cascade parents annevo_eval/data/humulus/mapping_PRJEB63534/{dl_gbs.sh,repair_dl.sh,runs.tsv} ENA FTP curl gzip -t + size > 100 kB only
Wye parents annevo_eval/scripts/download_PRJNA1357051.sh (copy in provenance_scripts/PRJNA1357051/) ENA filereport → curl -C - md5sum -c vs ENA fastq_md5
KR annevo_eval/data/humulus/_drafts/cordifolius_KR_reads/fetch_kr_genomic_reads.sh + kr_genomic_reads_manifest.tsv ENA FTP curl md5 vs ENA (10/10); KR/scripts/k01b_fastq_scan.sh read counts == ENA read_count
Carey 2026 CAR/PIPELINE.sh §0–1; CAR/scripts/lepton/sra_stagein_worker.sh, fastq_worker.sh, submit_carey.py NCBI SDL (locate.ncbi.nlm.nih.gov/sdl/2/retrieve?acc=<SRR>…) for size+md5 → on Lepton aws s3 cp --no-sign-request s3://sra-pub-run-odp/sra/<SRR>/<SRR> → fasterq-dump --split-3 --seq-defline '@$ac.$si $sn length=$rl' --qual-defline '+' bytes+md5 == SDL; records per mate == ENA read_count (exact); gzip -t; md5 recorded; FASTQ → AWS s3://bionemofw/users/skothenhill/carey2026/fastq/ and PBSS archive/carey2026_PRJNA1437418/fastq/

Where reads live now (PBSS): Apollo across two prefixes archive/PRJEB63565/reads/ + archive/hop_mapping_PRJEB63565_data2/, plus reads_repaired/ (ASSAYS 8241: take URIs from LL/sheets/apollo_reads_manifest.tsv, never a guessed prefix); archive/cascade_PRJEB63534_reads/, archive/wye_PRJNA1357051_reads/, archive/cordifolius_KR_reads/, archive/SW_reads/.

QC gate (required, from the gotchas): 1. Record count per mate == archive read_count (ASSAYS 3217). Bytes are a staging check only; ENA fastq_bytes ≠ staged size by 6–8 % (ASSAYS 5489). 2. For any .bz2→.gz rebuild, assert first header is @<ACC>. (ASSAYS 3371). 3. Repaired file must supersede the original in the manifest the drivers stage from (ASSAYS 5830; memory apollo-cross-truncated-runs). 4. Decode the true instrument from read headers (python3 annevo_eval/data_inventory/query.py --platform; Carey: CAR/scripts/build_sample_table.py prefix table). 5. Old Illumina (GA-II/IIx): check quality encoding; convert Phred+64 with KR/scripts/k06b_trim_requal.py (ASSAYS 8315).

Gotchas: ENA/ODP throughput from the annevo box is 0.02–0.6 MB/s; on Lepton ENA ~22 MB/s with ≤ 16 connections, ODP ~57 MB/s (ASSAYS 8491, 8211). EOS: transfer only via PBSS, never ENA on the cluster (CLAUDE.md). ENA publishes no read_count for ERR11641723 / ERR11641819 (ANCHOR_EXEMPT).

After download: annevo_eval/data_inventory/register_product.sh, entry in catalog.yaml, refresh.sh (CLAUDE.md; Carey has a catalog entry carey2026_wgs_reads).

S2. Read QC

S3. Alignment

Reference: 21110M male_ref.fa for everything the introgression atlas uses (Chr01–09, ChrX, ChrY; Chr01–05 + ChrX reverse-complemented vs drHumLupu-forward; TECH §1.1). Dosage additionally uses apollo_h1, apollo_h2, saaz_h1, hl1012_h1, cascade (PCD/PIPELINE.sh §0). Carey also aligned to drHumLupu (and MN1421/MN586 HAP1 for 8 wild runs, QC only).

Tool: GPU pbrun fq2bam (Parabricks nvcr.io/nvidia/clara/clara-parabricks:4.7.0-1), standing rule (memory use-parabricks-for-alignment). Needs the classic bwa index (.amb .ann .bwt .pac .sa), asserted .ann length == sum of .fai (ASSAYS 4.2f; LL/upload_refs.sh).

WGS duplicate rule (inverted vs GBS): markdup ON (omit --no-markdups; add --out-duplicate-metrics); downstream exclude dups (--ff UNMAP,SECONDARY,QCFAIL,DUP; depth -F 0xF04). Gate: samtools view -c -f 1024 <bam> > 0 for WGS, == 0 for GBS (LL/README.md "Five things" #2; ASSAYS 8271). Pooled invocations need a unique LB per sample, else cross-plant duplicates (ASSAYS 8281; job_ref.sh LB_CONTROL=1).

Read group: @RG\tID:<run>\tSM:<sample>\tPL:ILLUMINA\tPU:<run>\tLB:<sample or library>; PU is mandatory (memory use-parabricks-for-alignment). Multi-run samples (Pub_740M) = several --in-fq pairs in one call (align_sample in LL/linear_job.sh).

Which alignment fed what (this matters — they are different BAM sets):

Product Script Compute Key settings Output Fed
Apollo full-depth linear LL/linear_job.sh + LL/config/apollo_21110M.env, submit_linear.py apollo_21110M Lepton gpu.8xh100-sxm MARKDUP=1 PARENT_MARKDUP=1, GBS_BATCH=0, REPAIRED_URI/REPAIRED_RUNS, SIZE_MANIFEST, READCOUNT_MANIFEST, ANCHOR_EXEMPT PBSS lepton_linear/apollo_21110M/ (BAMs; QC 186/186, dup 7.22 %) joint VCFs (S4b); route-V baseline only. Not the atlas route-B BAMs (inferred from MCW/jobs/apollo_jobs.tsv)
WGS markdup set, 6 refs (<ref>_wgsmd) PCD/scripts/lepton/{make_bundle.sh,submit_pcd.py,job_ref.sh} Lepton, one 8×H100 job per ref --bundle wgsmd --cohorts "cascade wye apollo" --fastp-wgs-parents --env ONLY_WGS=1 MARKDUP_WGS=1 COUNTER=count_one_v2.sh FETCH_PROGENY=1 CALL_FF=UNMAP,SECONDARY,QCFAIL,DUP LB_CONTROL=1 RECOUNT_GBS=1; pooled --in-fq-list, split by RG, reheader SM. Apollo = FIRST 18,000,000 pairs per sample (~2×); father first 4.5 M pairs of each of 4 runs (00d_build_manifest.py, job_ref.sh sub_one) P/<ref>_wgsmd/bam/{apollo,cascade,wye}/<sample>.<ref>.bam atlas route B (apollo_bam, Cascade/Wye parents), lineage counts, sex pileups, dosage
KR / SW75 KR/scripts/k01d_align_kr_trimmed.sh, k06c_sw75_fastp_align.sh local 2× A6000, docker run --gpus all (bind mounts only under /data) --num-gpus 2, markdup ON, one RG per run, LB:DRX022141 /data/hop_data/cordifolius_KR_21110M/KR_cordifolius.21110M.bam route K, lineage, sex
Carey CAR/scripts/lepton/carey_gpu_worker.sh (submit_carey.py gpu) Lepton 8×H100 per sample, markdup ON, LB:<SRR>; refs from fs1 /mnt/fs1/skothenhill/linear/shared/ref_21110M/ AWS bionemofw/users/skothenhill/carey2026/bam/{21110M,drHumLupu}/ → PBSS via aws_bridge.sh to-pbss lineage counts + kt tags (in-job)
OSU deep (Cascade WGS) sync/2026-10-02/osu_21110M_genotyping/lepton/{deep_job.sh,submit_deep.py} Lepton markdup ON fs1 / PBSS osu_PRJNA1424574_21110M/deep/ OSU joint call + KING only

QC gates used: primary records (samtools view -c -F 0x900) == fastp output reads (Carey; KR); vs archive read count (READCOUNT_MANIFEST, linear_job.sh; memory verification-must-be-able-to-fail: BAM == fastp is self-referential); samtools flagstat; dup gate; mosdepth -n --fast-mode -F 1796 for measured mean depth (Carey). Per-sample QC table: CAR/scripts/build_sample_table.py → CAR/tables/carey2026_sample_qc.tsv (best template).

GENERIC: LL/linear_job.sh + one config/<pop>_<ref>.env + sheets/<pop>_sample_sheet.tsv (run → alias) + upload_inputs.sh + submit_linear.py. COPY-AND-EDIT: PCD/scripts/lepton/* (cohort list and read sources in 00d_build_manifest.py), CAR/scripts/lepton/* (sample sheet, refs, bucket prefix), KR/scripts/k01*. Compute site: all WGS alignment on Lepton (or local A6000 for KR). Nothing in the WGS records ran on EOS (inferred from records found). Lepton→PBSS is degraded; move data via AWS s3://bionemofw + annevo_eval/data_inventory/aws_bridge.sh (CLAUDE.md; ASSAYS 4.2e).

S4. Calling / allele counting

The atlas does not use genotype calls. It uses per-allele read counts (FORMAT/AD) at fixed site catalogues from per-sample BAM pileups.

S4a. Introgression base (clade A/B) counts — route B. MC/scripts/m01_count_one.sh <cohort> <sample> <pbss_key> WGS [targets], driver MC/scripts/m01_run_jobs.sh <jobs.tsv> 8 [targets]:

bcftools(1.20) mpileup -f $REF -T <targets> --min-MQ 20 --min-BQ 20 --ff UNMAP,SECONDARY,QCFAIL,DUP -a FORMAT/AD -Ou <presigned BAM url> \
 | bcftools query -f '%CHROM\t%POS\t%REF\t%ALT[\t%AD]\n'   -> MCW/bamcounts/<cohort>/<sample>.tsv.gz + .done   (FAIL if <= 1000 lines)

S4b. Joint VCF (not used by the served atlas for WGS cohorts). LL/config/apollo_21110M_untargeted.env: ROLE=call UNTARGETED=1 CALL_FF=UNMAP,SECONDARY,QCFAIL,DUP MIN_MQ_SET="0 20", joint call of 186 BAMs, no site list, no depth floor → PBSS lepton_linear/apollo_21110M_untargeted/calls/apollo_21110M_cohort_joint_mq{0,20}.vcf.gz (in that prefix, both/union = the superseded targeted design; trust only *_untargeted markers — memory apollo-cross-coverage-design). Used for route-V baseline Apollo (CI/PIPELINE.sh:44-45, 3 × 100 kb windows per 5 Mb) — superseded by route B. Gate rule: test an MQ floor by INFO/MQ minimum and depth, never by "mq20 < mq0" record counts (ASSAYS 5634).

S4c. Lineage SNP-layer counts. LR1/scripts/l02_count_one.sh <bam> <sample> <outdir> WGS (WGS ⇒ targets_wgs500.tsv.gz from l01b_wgs_targets.py, --ff …,DUP); job lists l02_make_wgs_jobs.py → L/jobs/{kr_wgs,cascade_wgs,wye_wgs,apollo_wgs}.tsv; driver LR1/scripts/l02_run_wgs.sh. Then LR1/scripts/l03_build_sparse.py <cohort> → L/sparse/<cohort>/. Carey ran the same mpileup per chromosome inside the GPU job (CAR/PIPELINE.sh §2) → L/bamcounts/carey2026/.

S4d. Tag-space counts (subspecies layer). samtools view -F 0xD00 <bam> | kt --mode scan --out <s>.kt --keep-loci L/tagspace/catalogue/keep_loci.u64 (WGS; GBS uses -F 0x900 … --mode gbs5scan); kt built from LR2/scripts/kt.cpp; driver LR2/scripts/kt_run_wgs.sh → LR2/scripts/t03_tag_sparse.py <cohort> 8 / t03_run.sh. Carey: in-job → L/tags/carey2026/.

S4e. Sex-chromosome pileups. SX/scripts/s10_pileup_one.sh via s10_run_jobs.sh $W/jobs/s10_jobs.tsv 8: bcftools mpileup -f $REF -T targets_sex.tsv -r ChrY,ChrX:225000001-268239420 --min-MQ 20 --min-BQ 20 --ff UNMAP,SECONDARY,QCFAIL,DUP -a FORMAT/AD -Ou "<bam url>##idx##<bai url>" → /data/hop_data/sex_chromosome_lineages/pileup/<cohort>/<sample>.tsv.gz, then s13_build_counts.py. Carey: not run.

S4f. Dosage read counts. In-job count_one_v2.sh BAM SAMPLE COHORT REFTAG WGS OUTDIR (PCD/scripts/lepton/): one pass over samtools view -F 0x900; primary / unmapped / dup / MAPQ<20 / MAPQ≥20 per sequence; WGS bins 100 kb of MAPQ≥20, non-dup, non-QCfail read starts (bins.tsv.gz) + any-MAPQ (binsall); FAIL if 0 primary records. Pulled with PCD/scripts/10_run_local_analysis.sh into /data/hop_data/per_chromosome_dosage_multiref/counts/<ref>/<pop>/ (WGS from <ref>_wgsmd). Allele balance for WGS: Apollo only from the existing full-depth hap1 call set (PCD/scripts/02_apollo_h1_existing_ab.sh); at ~2× no AB exists (ASSAYS 8251). COPY-AND-EDIT.

S5. NA/EU base calls (introgression atlas)

Route Used for WGS Scripts Notes
B (BAM pileup) Apollo (apollo_bam), Cascade/Wye WGS parents (inside cascade_bam/wye_bam) MC/scripts/m03_build_bam_cohort.py <cohort> [150] [5] → MCW/cohorts/<c>_bam.npz; m04_impute_multires.py <c>_bam (EM once at 5 Mb, params held at 0.5/1/2/5 Mb) → MCW/calls/<c>_bam_<res>_calls.tsv.gz, MCW/imputation/<c>_bam_params.json; m05_evaluate.py (must run before m06/m07) → m06/m06b/m07/m07b/m11; chain MC/scripts/m12_final_chain.sh nA/nB per catalogue A/B base; site kept if read in ≥ 5 samples; 150 bp grid thinning; cap 16 reads/site; τ 0.5; call if pmax ≥ 0.95 and ≥ 3 sites (TECH §3.2, §4). Apollo params e 0.0064, η 0.0121, dA 0.040, dB 0.109, w 0.853
K (borrowed params) KR, and Carey's lineage calls KR/scripts/k04a_parse_counts.py, k04b_kr_calls.py (+ k_model.py; verification reproduces 2 Apollo samples, must-fail with Wye params) → /data/hop_data/cordifolius_KR_21110M/calls/cordifolius_<res>_calls.tsv.gz single plant cannot fit EM; uses apollo_bam params
V (VCF AD) not used for WGS in the served atlas CI/scripts/07_extract_cohort.py, 08_impute.py variants-only VCFs create "depleted" bins (ASSAYS 8299) — the reason route B exists

COPY-AND-EDIT points for a new WGS cohort (route B): - m03 reads its sample list from CIW/cohorts/<coh>.npz (the route-V baseline) and asserts every sample has a .done; a cohort with no baseline needs the OSU fork pattern (/data/hop_data/osu_introgression_atlas/scripts/o03_build_bam_cohort.py, sample list from a text file). - m04: TRUTH, TRIP dicts (lines 43–57), CROSS_VCF; OSU fork o04_impute_multires.py adds cohort lines. - m05/m07: FINAL, parent map ("apollo": ("Apollo_mother","Pub_740M")), ASSAY label strings. - Mixed-assay call sets share one dA/dB (TECH I-20). Fit the EM on ≥ several samples; single plants → route K. QC gate: m05 truth accuracy (Apollo_mother is a truth plant vs Apollo h1/h2, scored on leave-plant-out sites), callability, parent-inconsistent rate; viewer warning thresholds callable < 0.90 / truth < 0.99 (TECH §4.5). KR has a row-specific test k21 (KR/scripts/viewer_tests/).

S6. Viewer export for the base atlas

No generator (TECH §10.2, §11.2). What was done: - Apollo: MC/scripts/m07_viewer_export.py → multires/atlas_{samples,native,transmission}_<res>.json; detail sync/2026-09-29/universal_introgression_atlas/scripts/u17_multires_detail.py → /data/hop_data/universal_introgression_atlas/viewer_detail_mr/ (TECH §10.1). - KR: KR/scripts/k08_viewer_export.py (group "cordifolius (KR) — WGS 75 bp", detail files, global similarity recomputed), tests KR/scripts/viewer_tests/run_tests.sh <res> + p08_test_summary.py. - New cohort pattern: OSU o00_labels.py → o07_viewer_export.py → o08_preset_patch.py → o17_detail_osu.py → o09_swap_into_served.sh, all under /data/hop_data/osu_introgression_atlas/scripts/; full 15-step sequence in TECH §10.2. COPY-AND-EDIT (parents, labels, PANEL_ORDER hard-coded). - Carey 2026 stopped here: no multires/atlas_samples_<res>.json entry, so the layer exports skip it (CAR/layers/README.md).

S7. Layers

Subspecies / lineage (hop_atlas/components/INTROGRESSION.md §4.3): 1. SNP layer: LR2/scripts/c01_call_levels_v6.py <cohort> --groups $GF --nproc 8 --tag _v6 → L/calls/<cohort>_v6/ (c01_run_all_v6.sh). Edit CALLSET and PARAMS dicts: WGS cohorts without their own fit borrow 'MCW/apollo_bam' (KR, Carey). 2. Tag layer: LR_ROOT=$L/tagspace python3 c01t_call_levels.py <cohort> --groups $GF --nproc 8 --minref 1 --tag _t4 --l0-from '$L/calls/{coh}_v6' → L/tagspace/calls/<cohort>_t4/. Takes sex and the ChrX L0 constraint from the served sexchrom JSON; cohorts absent from it are scored diploid on ChrX (Carey; CAR/layers/README.md). 3. Register: add the cohort to COH in e03_viewer_export.py, e04_detail_export.py, l23_lineage_v3_test.py, and to LIMITED_COH (WGS: European-lineage calls from GBS-trained groups are not shown; WGS parents inside GBS cohorts via LIMITED_KEYS). Carey did this as copies in CAR/layers/ used through ATLAS_LAYERS_EXPORT_DIR and ATLAS_LAYERS_CHECKS. 4. Export/test: hop_atlas/tools/atlas_layers.py lineage|test (S9). - Depth cap test for WGS lineage: LR2/scripts/x07_cap_test.sh, x08_cap_compare.py (cap 1/2/16).

Sex lanes (TECH §8; SX/PIPELINE.sh): s13_build_counts.py → s14_sex_table.py → s15_y_lineage.py (WGS plants in GBS cohorts and KR use the Apollo reference-female masks) → s16_x_caller.py --no-mask → s16b_hxr_male_mask.py → s16_x_caller.py (1 Mb X and 5 Mb yb MSY bins only for WGS plants) → s24_viewer_export.py (asserts exactly 1,272 samples; cannot regenerate the current file, TECH I-34) → s27_viewer_integrate.py. OSU extension pattern: o11_sexchrom_osu.py merges into the served JSON from a backup. Sex sources: annevo_eval/sample_identity/sample_sex_verified.tsv (Cascade WGS parents: "parent_library_X_test_not_applicable", sex = Y k-mers + role). KR sex = F by PI decision. COPY-AND-EDIT.

Phasing / concordance (PP/PIPELINE.sh): p01_phase_concordance.py <res> ×4 (hard-coded SRC, parent and ASM dicts, e.g. Apollo mother phase from Apollo_h1/h2 if genotype agreement ≥ 0.95), p03_bfreq_transmission.py, p05_validate_concordance.py, p04_export_viewer.sh; then ChrX sex-aware via hop_atlas/tools/atlas_layers.py xconc $STG (LR2/scripts/x11_sexaware_x_concordance.py). Crosses only. COPY-AND-EDIT (OSU fork o01_phase_osu.py).

Pedigree: atlas_layers.py pedigree $STG (GENERIC); name matching in LR2/scripts/p01_pedigree_build.py (SYN, UNVER, EXCL, CROSS); new cross = one CROSS line.

S8. Dosage atlas

Dependency order (hop_atlas/components/DOSAGE.md §2–3): S3 per reference (<ref>_wgsmd) → S4f counts → clade calls from S5 (build_cn.py reads MCW/calls/<CALLSRC[pop]>_1Mb_calls.tsv.gz; ancestry-aware CN needs them) → v1 assets (/data/hop_data/hop_atlas/dosage_sources/2026-09-28/dosage_explorer/scripts/build_all.py, build_assets.py <ref>) → v2 builder /data/hop_data/dosage_explorer_v2/stage/scripts/rebuild_and_validate.sh (build_all_cn.sh → worked_example.py → fill_defs.py → make_page.py → validate_cn.py) → hop_atlas/tools/relocate_dosage.py. - Hard-coded lists to edit: build_all_cn.sh (ref × pop loops), validate_cn.py and check_cn_browser.py (REFS, POPS), v1 build_all.py, mr_config.py (REFS), build_cn.py (CALLSRC, FRAME, REFHAP, EXCLUDE_BASE). - WGS specifics: 100 kb bins; duplicates excluded; Apollo depth from ~2× subsample; X not readable as copy number in Cascade/Wye/Apollo (memory per-chromosome-dosage). Pan-X instrument = annevo_eval/methods/sexchrom_dosage/hop_xdosage (count --input <bam>, call --manifest), not in the served atlas. - Generic package: annevo_eval/methods/autosomal_dosage/hop_adosage (cn --bins … --manifest … --assay …) — merged to main in commit 029bf81; DOSAGE.md still says "unmerged branch" (stale). - KR and Carey: not in the dosage atlas.

S9. Stage → build → test → deploy (GENERIC)

T=hop_atlas/tools/atlas_layers.py        # introgression layers (annevo_eval/scripts/atlas_layers.py is a 285-byte shim)
python3 $T stage --name <change>; STG=/data/hop_data/lineage_refinement/stages/<change>
python3 $T lineage $STG; python3 $T pedigree $STG; python3 $T xconc $STG
python3 $T serve $STG --port 8907; python3 $T test $STG --port 8907; python3 $T deploy $STG      # deploy needs PI approval
D=hop_atlas/tools/atlas_deploy.py        # dosage: stage dosage / build dosage / test dosage / deploy dosage [--dry-run]

Base-atlas JSON (S6) has no stage tool of its own; OSU used a hand-copied stage on :8901 and o09_swap_into_served.sh (md5-guarded). Rules: hop_atlas/README.md (never copy an old stage over the served copy; CHANGELOG entry; PI approval).


3. Per-cohort differences

Apollo KR Cascade parents Wye parents Carey 2026
Download NCBI prefetch (box); 10 truncated → .bz2 repair ENA curl + md5 ENA curl, gzip -t only ENA curl + md5 SRA ODP on Lepton + SDL md5 + read_count
Read quirks mother 35–265 bp; father 4 runs merged; 2 runs lack ENA read_count GA-IIx, all-N R2 reads 150 bp 250 bp → pbrun crash untrimmed; one ref needed 150 bp crop NovaSeq X / 6000
Atlas BAMs P/m21_wgsmd/bam/apollo/ (~2× subsample) local full depth P/m21_wgsmd/bam/cascade/ (full) P/m21_wgsmd/bam/wye/ (full) AWS/PBSS carey2026/bam/21110M/ (full)
Route-B targets 500 bp thinned (2.34 M) full catalogue per chromosome full catalogue full catalogue none (no base atlas)
EM params own apollo_bam fit borrowed apollo_bam shared with GBS progeny (cascade_bam) shared with GBS progeny (wye_bam) borrowed apollo_bam (lineage only)
Lineage layer LIMITED_COH LIMITED_COH (ATLAS_COH kr→cordifolius) LIMITED_KEYS LIMITED_KEYS LIMITED_COH (copies in CAR/layers/)
Sex MSY caller + Apollo masks, 1 Mb X + yb F (PI) Y k-mers + role; Apollo masks Apollo masks not run
Phasing progeny linkage + Apollo h1/h2 assembly n/a progeny linkage progeny linkage n/a (panel)
Dosage yes (6 refs) no (separate sync/2026-09-30/cordifolius_KR_multiref/) yes yes no

4. Same plant, different assay (WGS ↔ GBS/other)

Canonical table: annevo_eval/sample_identity/{sample_identity.tsv,identity_pairs.tsv,README.md}. Built by sync/2026-10-03/atlas_evidence_rebuild/scripts/e01_identity_table.py + e03_lineages.py (never hand-edit; re-run sync/2026-10-04/verified_sex_update/scripts/v01_identity_sex.py after). Same-plant groups (P####) = connected components of merged=True edges; status ok / duplicate / unresolved / non-progeny / different-plant / mislabel / mixture.

Naming: each library stays its own atlas row, keyed <cohort>|<sample> (WGS keys: run accession for Cascade parents and Carey; alias for Apollo/Wye/KR). Rows are never merged; identity is a group label, plus pedigree matching (p01_pedigree_build.py: exact / variant / synonym / approximate / unverified).

Identity checks used (records):

Pair Method Result Source
WGS cross parents vs panel GBS (Cascade, Apollo, WyeTarget) 5 Mb clade concordance + opposite-hom count 0.995–1.0, opp-hom 0 → merged identity_pairs.tsv (h02, sync/2026-09-30/haplotype_incompatibility/tables/pedigree_identity_pairs.tsv)
Apollo_mother vs Apollo h1/h2 assembly clade concordance; assembly provenance same plant same
OSU Cascade WGS (ERR11608298) vs OSU/USDA/divpanel Cascade GBS cross-call-set KING-robust + IBS0 on shared records, both DP ≥ 8 (osu_population_qc/scripts/10_crossdata_identity.py, 14_crossdata_tables.py); rule same ≥ 0.354 & IBS0 ≤ 0.005 GBS-vs-WGS 0.231 "ambiguous (GBS-vs-WGS attenuation)"; WGS vs USDA 0.387 same sync/2026-10-02/osu_population_qc/REPORT.md
same, within one joint call (GBS + deep) plink2 KING at DP ≥ 15, MAF ≥ 0.05 0.387 / 0.420 sync/2026-10-02/osu_21110M_genotyping/README.md:106
transmitted-allele test fraction of a sample's sites carrying alleles the Cascade mother transmitted WGS 0.877 rank 1 osu_population_qc/scripts/13_cascade_transmitted.py
lineage-layer agreement by assay pairing same-plant pairs, SNP v5 vs tag, bins identical tables only LR2/scripts/x03_cross_assay.py → LR2/tables/X03_cross_assay_{pairs,summary}.tsv; x05_pair_crosstab.py cascade ERR11608298 osu cascade

Recorded caveats: KING thresholds calibrated on GBS–GBS pairs; GBS-vs-deep pairs fall below them (REPORT calibration note). Cascade WGS mother vs OSU Cascade GBS is unresolved in sample_identity.tsv. Carey overlaps are "recorded, not resolved": Carey Cascade SRR37629444 is a different library from ERR11608298; Carey 19058M possibly = USDA 19058_M (DATA_CATALOG carey2026_wgs_reads). The OSU cross uses WGS ERR11608298 as mother in osu2022005_21110M, flagged SUPERSEDED (mother = WGS) in the explorer (sync/2026-10-02/osu_explorer_update/README.md:25); the served OSU atlas uses the OSU GBS cascade library as mother (TECH §1.3).


5. Missing or ambiguous steps

  1. No base-atlas generator (S6). Every WGS cohort since Apollo entered by a fork (KR k08, OSU o07). Carey is blocked here.
  2. Route-B job tables and the Apollo 500 bp target list have no producer script (MCW/jobs/*.tsv, targets_apollo_500bp.tsv.gz).
  3. Apollo atlas rows use the ~2× subsample BAMs, not the ~9× linear BAMs. No record says this was chosen for the atlas; it follows from MCW/jobs/apollo_jobs.tsv → m21_wgsmd + PCD/scripts/00d_build_manifest.py (inferred). Full-depth BAMs exist under PBSS lepton_linear/apollo_21110M/ (inferred path from LL PUSH_PREFIX defaults).
  4. EM refit when adding samples to an existing cohort: undocumented (ADDING_DATA.md).
  5. Sex lane for a new cohort: s24 asserts 1,272 samples; only the OSU merge pattern exists. Not run for Carey.
  6. Dosage add-cohort: no procedure; ~6 hard-coded lists; WGS allele balance needs a full-depth call set.
  7. Download/QC: no generic tool; integrity checks differ per cohort (Cascade parents: gzip -t only — record count vs ENA never recorded for them).
  8. Stale/contradictory records: sample_identity.tsv row kr:KR lists run_accession DRR024404–407 (the SW75 runs), not KR's DRR024452–456. DOSAGE.md says hop_adosage is unmerged (merged in 029bf81). ADDING_DATA.md/INTROGRESSION.md cite cluster/lepton_linear correctly but the task brief's cluster/lepton_linear does not exist. TECH §10.2 says the OSU sync scripts folder is empty (it is not).
  9. Instrument for Cascade WGS parents never decoded from headers in any record found.
  10. Carey status: 15/22 runs FAILED/absent in the 2026-10-08 19:07 QC table; whether a re-run is in flight is not recorded in the folder (check CAR/scripts/lepton/poll_carey.sh output / Lepton job state before acting).
  11. Mixed-assay EM (GBS progeny + WGS parents share dA/dB) is a known formalization issue, not resolved (TECH I-20).

6. Best scripts to generalise (by stage)

Stage Start from Why
Download + QC CAR/scripts/lepton/fastq_worker.sh + sra_stagein_worker.sh SDL md5 + exact read_count gate + ODP speed; ENA-form headers
Align LL/linear_job.sh (config-only, resumable, dup gate, read-count anchor) already GENERIC; add markdup WGS defaults, per-ref loop from PCD/job_ref.sh
Per-sample bundle CAR/scripts/lepton/carey_gpu_worker.sh one job does align ×2 refs, QC, mosdepth, L02 counts, kt tags, push
QC table CAR/scripts/build_sample_table.py measured depth, decoded instrument, anchors
Counts MC/scripts/m01_count_one.sh, LR1/scripts/l02_count_one.sh, SX/scripts/s10_pileup_one.sh, PCD/scripts/lepton/count_one_v2.sh already CLI-shaped; same mpileup family, differ only in target list / region
Calls o03/o04 (OSU forks of m03/m04) show exactly which constants a new cohort needs
Export OSU o07/o08/o17/o09 + TECH §10.2 the only complete add-a-cohort sequence
Layers/deploy hop_atlas/tools/atlas_layers.py, atlas_deploy.py GENERIC; needs a cohort manifest in place of COH/LIMITED_COH