Hop atlas home · Overview · Changelog · Architecture · Adding_Data · Introgression · Ribbon · Dosage · Phylogeny · Bundle · Ingest · Ingest_Gbs · Ingest_Wgs · Ingest_Grasdi · Ingest_Rnaseq · Ingest_Assembly
Rendered from hop_atlas/ingest/WGS.md (2026-10-13)
Written: 2026-10-08. Status: reconstruction of what was actually run. No command below is new; every one is
quoted from, or points at, a dated record. Nothing here is a tested end-to-end pipeline (see ../ADDING_DATA.md).
Labels used:
- GENERIC: parameterised, reusable as is (config/env/CLI argument).
- COPY-AND-EDIT: a dated, cohort-specific script; copy it to your own dated folder and edit the named constants.
- RECORD: documentation only ("not meant to be re-executed blindly"); read it, then write your own job.
- (inferred): my reading of the records, not stated in them.
Path shorthands (same as annevo_eval/docs/INTROGRESSION_ATLAS_TECHNICAL.md, "TECH" below):
MC = sync/2026-09-30/multires_clade, MCW = /data/hop_data/multires_clade, CI = sync/2026-09-29/clade_ab_imputation,
CIW = /data/hop_data/clade_ab_imputation, PCD = sync/2026-09-25/per_chromosome_dosage_multiref,
P = s3://general-purpose/skothenhill/archive/per_chromosome_dosage_multiref (PBSS), LR1 = sync/2026-10-06/lineage_refinement,
LR2 = sync/2026-10-07/lineage_refinement_v2, L = /data/hop_data/lineage_refinement, SX = sync/2026-09-30/sex_chromosome_lineages,
PP = sync/2026-09-30/phased_parents_switch_concordance, KR = sync/2026-09-30/cordifolius_KR, CAR = sync/2026-10-08/carey2026_hop_wgs,
LL = cluster/lepton_linear (repo root, not annevo_eval/cluster/). REF = /data/hop_data/ref_21110M_male/male_ref.fa (21110M, Akagi numbering).
Untracked in git (2026-10-08 check): LL/, MC/, KR/, PCD/, CAR/. Only LR2/PIPELINE.sh and hop_atlas/tools/ are tracked.
| Cohort / samples | Accessions | Labelled vs true platform | Depth | Used for | Atlas status |
|---|---|---|---|---|---|
Apollo × Pub_740M (apollo, 186 rows) |
PRJEB63565. Mother ERR11639885 → Apollo_mother; father ERR11639886–889 (4 runs) → Pub_740M; progeny ERR11641697… → AxP_234_### (sheet LL/sheets/apollo_sample_sheet.tsv) |
Mother HiSeq 2500 (D00553R), reads 35–265 bp; father 4 lanes of one NovaSeq; progeny headers @A00275 (NovaSeq) (LL/config/apollo_21110M.env; ASSAYS 3371). RNA-seq/Hi-C of the same project are mislabelled (memory sequencing-platform-labels-unreliable) |
progeny ~9×, mother ~30×, father ~90× (memory apollo-cross-coverage-design). Atlas BAMs are a ~2× subsample (below) |
base NA/EU (route B apollo_bam), lineage/subspecies (LIMITED), sex lanes (X/Y, WGS 1 Mb + yb), phasing (Apollo h1/h2 assembly phase), dosage (all 6 refs) |
served |
cordifolius KR (cordifolius, 1 row) |
PRJDB3233, DRR024452–456 | GA-IIx (HWUSI-EAS1725), 2×75 bp, Phred+33 (KR/PIPELINE.sh) |
~8× (memory kr-multiref-competitive-mapping) |
base NA/EU (route K, borrowed params), lineage (LIMITED), sex lane (KR = F, PI decision). Not in dosage atlas | served |
| SW75 Shinshu Wase (control for KR) | DRR024404–407 | GA-IIx, 2×109 bp, Phred+64 (ASSAYS 8315) | — | KR comparator; lineage counts (kr_wgs jobs) |
not a served row (inferred) |
Cascade × LB_male parents (WGS rows inside GBS cohort cascade) |
PRJEB63534: ERR11608298 (Cascade ♀), ERR11608479 (LB_male ♂; is the father, memory cascade-cross-father-is-present) |
150 bp PE (ASSAYS 8261); instrument not decoded in any record found | 259 M / 211 M reads; OSU deep QC: autosomes ~12×, ChrX 25× for ERR11608298 (sync/2026-10-02/osu_21110M_genotyping/README.md:99) |
parents of cascade_bam, sex lanes, phasing, dosage; also used as the OSU cross "deep mother" |
served |
Wye Target × BL2/1 parents (WGS rows inside wye) |
PRJNA1357051: SRR35941433 → WyeTarget, SRR35941434 → BL2_1 |
labelled NovaSeq 6000, 250 bp PE (LL/config/wye_21110M.env) |
8.3× / 9.0× raw; measured 3.5–4.7× at MQ≥20 (DATA_CATALOG Wye entry) | parents of wye_bam, sex lanes, phasing, dosage |
served |
Carey 2026 (carey2026) |
PRJNA1437418, 22 runs (/data/hop_data/carey2026_PRJNA1437418/carey2026_sample_sheet.tsv) |
decoded per run from headers: NovaSeq 6000 / NovaSeq X Plus (CAR/tables/carey2026_sample_qc.tsv) |
6.8–10.8× autosomal on 21110M (7 runs done) | lineage/subspecies calls computed; no base-atlas rows; not deployed | in progress: QC table (2026-10-08 19:07) lists 7 SAMPLE_OK, 15 FAILED/absent |
| OSU deep samples (not atlas rows) | Cascade WGS ERR11608298 re-aligned; Hi-C SRR37256446/7 treated as deep | — | — | OSU joint GBS+deep call set, KING identity (sync/2026-10-02/osu_21110M_genotyping/PIPELINE.sh §9) |
not an atlas route |
No WGS exists in our records for: Havill parents (GBS, 2 libraries each), divpanel / USDA / OSU panels (GBS),
Saaz × HL1012 (GRAS-Di). The "comparative WGS" set (PMR-480/MN-101 etc.) was deleted 2026-10-02 (DATA_CATALOG comparative_wgs).
| Question | Why it matters | Where it was decided |
|---|---|---|
| Cross (parents + progeny) or panel (no parents)? | phasing, transmission, NO_PARENTS=1, truth plants |
LL/linear_job.sh header; TECH §1.3 |
| All WGS, or WGS parents inside a GBS cohort? | dup rule per sample; EM parameters are shared per call set across assays (TECH §4.2, issue I-20) | MC/PIPELINE.sh |
| Single plant? | route K (borrowed parameters) | KR/ |
| Low-pass (≤ ~10×) progeny? | calling must be joint + untargeted, no depth floors (memory apollo-cross-coverage-design) |
LL/config/apollo_21110M_untargeted.env |
Records (all RECORD / COPY-AND-EDIT; no generic downloader exists):
| Cohort | Script | Source / tool | Integrity check used |
|---|---|---|---|
| Apollo | annevo_eval/data_inventory/provenance_scripts/PRJEB63565/ncbi_worker.sh (+ watchdog6.sh) |
NCBI prefetch --max-size 200g → fasterq-dump --split-files --threads 4 → pigz (EBI throttled to ~0.85 MB/s) |
none adequate: gzip -t only → 10 truncated runs (below) |
| Apollo repair | sync/2026-09-05/lepton_linear/APOLLO_TRUNCATED_RUNS.md |
submitter .bz2 → .gz with ENA headers rebuilt; PBSS PRJEB63565/reads_repaired/ |
bytes/md5 vs ENA submitted_*; records == ENA read_count/2; header form @<ACC>. |
| Cascade parents | annevo_eval/data/humulus/mapping_PRJEB63534/{dl_gbs.sh,repair_dl.sh,runs.tsv} |
ENA FTP curl |
gzip -t + size > 100 kB only |
| Wye parents | annevo_eval/scripts/download_PRJNA1357051.sh (copy in provenance_scripts/PRJNA1357051/) |
ENA filereport → curl -C - |
md5sum -c vs ENA fastq_md5 |
| KR | annevo_eval/data/humulus/_drafts/cordifolius_KR_reads/fetch_kr_genomic_reads.sh + kr_genomic_reads_manifest.tsv |
ENA FTP curl |
md5 vs ENA (10/10); KR/scripts/k01b_fastq_scan.sh read counts == ENA read_count |
| Carey 2026 | CAR/PIPELINE.sh §0–1; CAR/scripts/lepton/sra_stagein_worker.sh, fastq_worker.sh, submit_carey.py |
NCBI SDL (locate.ncbi.nlm.nih.gov/sdl/2/retrieve?acc=<SRR>…) for size+md5 → on Lepton aws s3 cp --no-sign-request s3://sra-pub-run-odp/sra/<SRR>/<SRR> → fasterq-dump --split-3 --seq-defline '@$ac.$si $sn length=$rl' --qual-defline '+' |
bytes+md5 == SDL; records per mate == ENA read_count (exact); gzip -t; md5 recorded; FASTQ → AWS s3://bionemofw/users/skothenhill/carey2026/fastq/ and PBSS archive/carey2026_PRJNA1437418/fastq/ |
Where reads live now (PBSS): Apollo across two prefixes archive/PRJEB63565/reads/ + archive/hop_mapping_PRJEB63565_data2/,
plus reads_repaired/ (ASSAYS 8241: take URIs from LL/sheets/apollo_reads_manifest.tsv, never a guessed prefix);
archive/cascade_PRJEB63534_reads/, archive/wye_PRJNA1357051_reads/, archive/cordifolius_KR_reads/, archive/SW_reads/.
QC gate (required, from the gotchas):
1. Record count per mate == archive read_count (ASSAYS 3217). Bytes are a staging check only; ENA fastq_bytes ≠ staged size by 6–8 % (ASSAYS 5489).
2. For any .bz2→.gz rebuild, assert first header is @<ACC>. (ASSAYS 3371).
3. Repaired file must supersede the original in the manifest the drivers stage from (ASSAYS 5830; memory apollo-cross-truncated-runs).
4. Decode the true instrument from read headers (python3 annevo_eval/data_inventory/query.py --platform; Carey: CAR/scripts/build_sample_table.py prefix table).
5. Old Illumina (GA-II/IIx): check quality encoding; convert Phred+64 with KR/scripts/k06b_trim_requal.py (ASSAYS 8315).
Gotchas: ENA/ODP throughput from the annevo box is 0.02–0.6 MB/s; on Lepton ENA ~22 MB/s with ≤ 16 connections, ODP ~57 MB/s (ASSAYS 8491, 8211). EOS: transfer only via PBSS, never ENA on the cluster (CLAUDE.md). ENA publishes no read_count for ERR11641723 / ERR11641819 (ANCHOR_EXEMPT).
After download: annevo_eval/data_inventory/register_product.sh, entry in catalog.yaml, refresh.sh (CLAUDE.md; Carey has a catalog entry carey2026_wgs_reads).
fastp --detect_adapter_for_pe --length_required 30 --thread N --json <acc>.fastp.json (LL/linear_job.sh trim_one, FASTP_MIN_LEN=30; KR/scripts/k01c_fastp_kr.sh adds -w 4 -z 3; CAR/PIPELINE.sh §2).pbrun fq2bam 4.7.0-1 crashes with "illegal memory access" on untrimmed 250 bp PE (Wye parents) and raw 2×75 GA-IIx (KR); fastp first fixed both (ASSAYS 8261, 8307). One Wye parent on Apollo h2 additionally needed cropping fastp -b 150 -B 150 -A -G -Q -L (PCD/PIPELINE.sh).before_filtering.total_reads == 2 × archive read_count (Carey; ASSAYS 5489 "the real completeness gate"). fastp aborting mid-file is how truncation was finally caught (memory apollo-cross-truncated-runs).LL/linear_job.sh (DO_FASTP=1 default). No standalone QC tool.Reference: 21110M male_ref.fa for everything the introgression atlas uses (Chr01–09, ChrX, ChrY; Chr01–05 + ChrX reverse-complemented vs drHumLupu-forward; TECH §1.1). Dosage additionally uses apollo_h1, apollo_h2, saaz_h1, hl1012_h1, cascade (PCD/PIPELINE.sh §0). Carey also aligned to drHumLupu (and MN1421/MN586 HAP1 for 8 wild runs, QC only).
Tool: GPU pbrun fq2bam (Parabricks nvcr.io/nvidia/clara/clara-parabricks:4.7.0-1), standing rule (memory use-parabricks-for-alignment). Needs the classic bwa index (.amb .ann .bwt .pac .sa), asserted .ann length == sum of .fai (ASSAYS 4.2f; LL/upload_refs.sh).
WGS duplicate rule (inverted vs GBS): markdup ON (omit --no-markdups; add --out-duplicate-metrics); downstream exclude dups (--ff UNMAP,SECONDARY,QCFAIL,DUP; depth -F 0xF04). Gate: samtools view -c -f 1024 <bam> > 0 for WGS, == 0 for GBS (LL/README.md "Five things" #2; ASSAYS 8271). Pooled invocations need a unique LB per sample, else cross-plant duplicates (ASSAYS 8281; job_ref.sh LB_CONTROL=1).
Read group: @RG\tID:<run>\tSM:<sample>\tPL:ILLUMINA\tPU:<run>\tLB:<sample or library>; PU is mandatory (memory use-parabricks-for-alignment). Multi-run samples (Pub_740M) = several --in-fq pairs in one call (align_sample in LL/linear_job.sh).
Which alignment fed what (this matters — they are different BAM sets):
| Product | Script | Compute | Key settings | Output | Fed |
|---|---|---|---|---|---|
| Apollo full-depth linear | LL/linear_job.sh + LL/config/apollo_21110M.env, submit_linear.py apollo_21110M |
Lepton gpu.8xh100-sxm |
MARKDUP=1 PARENT_MARKDUP=1, GBS_BATCH=0, REPAIRED_URI/REPAIRED_RUNS, SIZE_MANIFEST, READCOUNT_MANIFEST, ANCHOR_EXEMPT |
PBSS lepton_linear/apollo_21110M/ (BAMs; QC 186/186, dup 7.22 %) |
joint VCFs (S4b); route-V baseline only. Not the atlas route-B BAMs (inferred from MCW/jobs/apollo_jobs.tsv) |
WGS markdup set, 6 refs (<ref>_wgsmd) |
PCD/scripts/lepton/{make_bundle.sh,submit_pcd.py,job_ref.sh} |
Lepton, one 8×H100 job per ref | --bundle wgsmd --cohorts "cascade wye apollo" --fastp-wgs-parents --env ONLY_WGS=1 MARKDUP_WGS=1 COUNTER=count_one_v2.sh FETCH_PROGENY=1 CALL_FF=UNMAP,SECONDARY,QCFAIL,DUP LB_CONTROL=1 RECOUNT_GBS=1; pooled --in-fq-list, split by RG, reheader SM. Apollo = FIRST 18,000,000 pairs per sample (~2×); father first 4.5 M pairs of each of 4 runs (00d_build_manifest.py, job_ref.sh sub_one) |
P/<ref>_wgsmd/bam/{apollo,cascade,wye}/<sample>.<ref>.bam |
atlas route B (apollo_bam, Cascade/Wye parents), lineage counts, sex pileups, dosage |
| KR / SW75 | KR/scripts/k01d_align_kr_trimmed.sh, k06c_sw75_fastp_align.sh |
local 2× A6000, docker run --gpus all (bind mounts only under /data) |
--num-gpus 2, markdup ON, one RG per run, LB:DRX022141 |
/data/hop_data/cordifolius_KR_21110M/KR_cordifolius.21110M.bam |
route K, lineage, sex |
| Carey | CAR/scripts/lepton/carey_gpu_worker.sh (submit_carey.py gpu) |
Lepton 8×H100 | per sample, markdup ON, LB:<SRR>; refs from fs1 /mnt/fs1/skothenhill/linear/shared/ref_21110M/ |
AWS bionemofw/users/skothenhill/carey2026/bam/{21110M,drHumLupu}/ → PBSS via aws_bridge.sh to-pbss |
lineage counts + kt tags (in-job) |
| OSU deep (Cascade WGS) | sync/2026-10-02/osu_21110M_genotyping/lepton/{deep_job.sh,submit_deep.py} |
Lepton | markdup ON | fs1 / PBSS osu_PRJNA1424574_21110M/deep/ |
OSU joint call + KING only |
QC gates used: primary records (samtools view -c -F 0x900) == fastp output reads (Carey; KR); vs archive read count (READCOUNT_MANIFEST, linear_job.sh; memory verification-must-be-able-to-fail: BAM == fastp is self-referential); samtools flagstat; dup gate; mosdepth -n --fast-mode -F 1796 for measured mean depth (Carey). Per-sample QC table: CAR/scripts/build_sample_table.py → CAR/tables/carey2026_sample_qc.tsv (best template).
GENERIC: LL/linear_job.sh + one config/<pop>_<ref>.env + sheets/<pop>_sample_sheet.tsv (run → alias) + upload_inputs.sh + submit_linear.py. COPY-AND-EDIT: PCD/scripts/lepton/* (cohort list and read sources in 00d_build_manifest.py), CAR/scripts/lepton/* (sample sheet, refs, bucket prefix), KR/scripts/k01*.
Compute site: all WGS alignment on Lepton (or local A6000 for KR). Nothing in the WGS records ran on EOS (inferred from records found). Lepton→PBSS is degraded; move data via AWS s3://bionemofw + annevo_eval/data_inventory/aws_bridge.sh (CLAUDE.md; ASSAYS 4.2e).
The atlas does not use genotype calls. It uses per-allele read counts (FORMAT/AD) at fixed site catalogues from per-sample BAM pileups.
S4a. Introgression base (clade A/B) counts — route B. MC/scripts/m01_count_one.sh <cohort> <sample> <pbss_key> WGS [targets], driver MC/scripts/m01_run_jobs.sh <jobs.tsv> 8 [targets]:
bcftools(1.20) mpileup -f $REF -T <targets> --min-MQ 20 --min-BQ 20 --ff UNMAP,SECONDARY,QCFAIL,DUP -a FORMAT/AD -Ou <presigned BAM url> \
| bcftools query -f '%CHROM\t%POS\t%REF\t%ALT[\t%AD]\n' -> MCW/bamcounts/<cohort>/<sample>.tsv.gz + .done (FAIL if <= 1000 lines)
CIW/diag/targets_all.tsv.gz (29.2 M sites; Cascade/Wye WGS parents) or the 500 bp thinned MCW/jobs/targets_apollo_500bp.tsv.gz (Apollo, 2.34 M sites, "to bound output size"). No script generates the job tables (MCW/jobs/cw_jobs.tsv, apollo_jobs.tsv) or targets_apollo_500bp.tsv.gz (TECH §1.2).KR/scripts/k02_pileup_catalog.sh <sample>, same flags, per chromosome (-r <chr>), full catalogue.-d 250, BAQ on, anomalous pairs skipped) are not overridden (TECH issue I-21). Supplementary alignments still pass --ff UNMAP,SECONDARY,QCFAIL (ASSAYS 7924).REF, BCF and output root.S4b. Joint VCF (not used by the served atlas for WGS cohorts). LL/config/apollo_21110M_untargeted.env: ROLE=call UNTARGETED=1 CALL_FF=UNMAP,SECONDARY,QCFAIL,DUP MIN_MQ_SET="0 20", joint call of 186 BAMs, no site list, no depth floor → PBSS lepton_linear/apollo_21110M_untargeted/calls/apollo_21110M_cohort_joint_mq{0,20}.vcf.gz (in that prefix, both/union = the superseded targeted design; trust only *_untargeted markers — memory apollo-cross-coverage-design). Used for route-V baseline Apollo (CI/PIPELINE.sh:44-45, 3 × 100 kb windows per 5 Mb) — superseded by route B. Gate rule: test an MQ floor by INFO/MQ minimum and depth, never by "mq20 < mq0" record counts (ASSAYS 5634).
S4c. Lineage SNP-layer counts. LR1/scripts/l02_count_one.sh <bam> <sample> <outdir> WGS (WGS ⇒ targets_wgs500.tsv.gz from l01b_wgs_targets.py, --ff …,DUP); job lists l02_make_wgs_jobs.py → L/jobs/{kr_wgs,cascade_wgs,wye_wgs,apollo_wgs}.tsv; driver LR1/scripts/l02_run_wgs.sh. Then LR1/scripts/l03_build_sparse.py <cohort> → L/sparse/<cohort>/. Carey ran the same mpileup per chromosome inside the GPU job (CAR/PIPELINE.sh §2) → L/bamcounts/carey2026/.
S4d. Tag-space counts (subspecies layer). samtools view -F 0xD00 <bam> | kt --mode scan --out <s>.kt --keep-loci L/tagspace/catalogue/keep_loci.u64 (WGS; GBS uses -F 0x900 … --mode gbs5scan); kt built from LR2/scripts/kt.cpp; driver LR2/scripts/kt_run_wgs.sh → LR2/scripts/t03_tag_sparse.py <cohort> 8 / t03_run.sh. Carey: in-job → L/tags/carey2026/.
S4e. Sex-chromosome pileups. SX/scripts/s10_pileup_one.sh via s10_run_jobs.sh $W/jobs/s10_jobs.tsv 8:
bcftools mpileup -f $REF -T targets_sex.tsv -r ChrY,ChrX:225000001-268239420 --min-MQ 20 --min-BQ 20 --ff UNMAP,SECONDARY,QCFAIL,DUP -a FORMAT/AD -Ou "<bam url>##idx##<bai url>" → /data/hop_data/sex_chromosome_lineages/pileup/<cohort>/<sample>.tsv.gz, then s13_build_counts.py. Carey: not run.
S4f. Dosage read counts. In-job count_one_v2.sh BAM SAMPLE COHORT REFTAG WGS OUTDIR (PCD/scripts/lepton/): one pass over samtools view -F 0x900; primary / unmapped / dup / MAPQ<20 / MAPQ≥20 per sequence; WGS bins 100 kb of MAPQ≥20, non-dup, non-QCfail read starts (bins.tsv.gz) + any-MAPQ (binsall); FAIL if 0 primary records. Pulled with PCD/scripts/10_run_local_analysis.sh into /data/hop_data/per_chromosome_dosage_multiref/counts/<ref>/<pop>/ (WGS from <ref>_wgsmd). Allele balance for WGS: Apollo only from the existing full-depth hap1 call set (PCD/scripts/02_apollo_h1_existing_ab.sh); at ~2× no AB exists (ASSAYS 8251). COPY-AND-EDIT.
| Route | Used for WGS | Scripts | Notes |
|---|---|---|---|
| B (BAM pileup) | Apollo (apollo_bam), Cascade/Wye WGS parents (inside cascade_bam/wye_bam) |
MC/scripts/m03_build_bam_cohort.py <cohort> [150] [5] → MCW/cohorts/<c>_bam.npz; m04_impute_multires.py <c>_bam (EM once at 5 Mb, params held at 0.5/1/2/5 Mb) → MCW/calls/<c>_bam_<res>_calls.tsv.gz, MCW/imputation/<c>_bam_params.json; m05_evaluate.py (must run before m06/m07) → m06/m06b/m07/m07b/m11; chain MC/scripts/m12_final_chain.sh |
nA/nB per catalogue A/B base; site kept if read in ≥ 5 samples; 150 bp grid thinning; cap 16 reads/site; τ 0.5; call if pmax ≥ 0.95 and ≥ 3 sites (TECH §3.2, §4). Apollo params e 0.0064, η 0.0121, dA 0.040, dB 0.109, w 0.853 |
| K (borrowed params) | KR, and Carey's lineage calls | KR/scripts/k04a_parse_counts.py, k04b_kr_calls.py (+ k_model.py; verification reproduces 2 Apollo samples, must-fail with Wye params) → /data/hop_data/cordifolius_KR_21110M/calls/cordifolius_<res>_calls.tsv.gz |
single plant cannot fit EM; uses apollo_bam params |
| V (VCF AD) | not used for WGS in the served atlas | CI/scripts/07_extract_cohort.py, 08_impute.py |
variants-only VCFs create "depleted" bins (ASSAYS 8299) — the reason route B exists |
COPY-AND-EDIT points for a new WGS cohort (route B):
- m03 reads its sample list from CIW/cohorts/<coh>.npz (the route-V baseline) and asserts every sample has a .done; a cohort with no baseline needs the OSU fork pattern (/data/hop_data/osu_introgression_atlas/scripts/o03_build_bam_cohort.py, sample list from a text file).
- m04: TRUTH, TRIP dicts (lines 43–57), CROSS_VCF; OSU fork o04_impute_multires.py adds cohort lines.
- m05/m07: FINAL, parent map ("apollo": ("Apollo_mother","Pub_740M")), ASSAY label strings.
- Mixed-assay call sets share one dA/dB (TECH I-20). Fit the EM on ≥ several samples; single plants → route K.
QC gate: m05 truth accuracy (Apollo_mother is a truth plant vs Apollo h1/h2, scored on leave-plant-out sites), callability, parent-inconsistent rate; viewer warning thresholds callable < 0.90 / truth < 0.99 (TECH §4.5). KR has a row-specific test k21 (KR/scripts/viewer_tests/).
No generator (TECH §10.2, §11.2). What was done:
- Apollo: MC/scripts/m07_viewer_export.py → multires/atlas_{samples,native,transmission}_<res>.json; detail sync/2026-09-29/universal_introgression_atlas/scripts/u17_multires_detail.py → /data/hop_data/universal_introgression_atlas/viewer_detail_mr/ (TECH §10.1).
- KR: KR/scripts/k08_viewer_export.py (group "cordifolius (KR) — WGS 75 bp", detail files, global similarity recomputed), tests KR/scripts/viewer_tests/run_tests.sh <res> + p08_test_summary.py.
- New cohort pattern: OSU o00_labels.py → o07_viewer_export.py → o08_preset_patch.py → o17_detail_osu.py → o09_swap_into_served.sh, all under /data/hop_data/osu_introgression_atlas/scripts/; full 15-step sequence in TECH §10.2. COPY-AND-EDIT (parents, labels, PANEL_ORDER hard-coded).
- Carey 2026 stopped here: no multires/atlas_samples_<res>.json entry, so the layer exports skip it (CAR/layers/README.md).
Subspecies / lineage (hop_atlas/components/INTROGRESSION.md §4.3):
1. SNP layer: LR2/scripts/c01_call_levels_v6.py <cohort> --groups $GF --nproc 8 --tag _v6 → L/calls/<cohort>_v6/ (c01_run_all_v6.sh). Edit CALLSET and PARAMS dicts: WGS cohorts without their own fit borrow 'MCW/apollo_bam' (KR, Carey).
2. Tag layer: LR_ROOT=$L/tagspace python3 c01t_call_levels.py <cohort> --groups $GF --nproc 8 --minref 1 --tag _t4 --l0-from '$L/calls/{coh}_v6' → L/tagspace/calls/<cohort>_t4/. Takes sex and the ChrX L0 constraint from the served sexchrom JSON; cohorts absent from it are scored diploid on ChrX (Carey; CAR/layers/README.md).
3. Register: add the cohort to COH in e03_viewer_export.py, e04_detail_export.py, l23_lineage_v3_test.py, and to LIMITED_COH (WGS: European-lineage calls from GBS-trained groups are not shown; WGS parents inside GBS cohorts via LIMITED_KEYS). Carey did this as copies in CAR/layers/ used through ATLAS_LAYERS_EXPORT_DIR and ATLAS_LAYERS_CHECKS.
4. Export/test: hop_atlas/tools/atlas_layers.py lineage|test (S9).
- Depth cap test for WGS lineage: LR2/scripts/x07_cap_test.sh, x08_cap_compare.py (cap 1/2/16).
Sex lanes (TECH §8; SX/PIPELINE.sh): s13_build_counts.py → s14_sex_table.py → s15_y_lineage.py (WGS plants in GBS cohorts and KR use the Apollo reference-female masks) → s16_x_caller.py --no-mask → s16b_hxr_male_mask.py → s16_x_caller.py (1 Mb X and 5 Mb yb MSY bins only for WGS plants) → s24_viewer_export.py (asserts exactly 1,272 samples; cannot regenerate the current file, TECH I-34) → s27_viewer_integrate.py. OSU extension pattern: o11_sexchrom_osu.py merges into the served JSON from a backup. Sex sources: annevo_eval/sample_identity/sample_sex_verified.tsv (Cascade WGS parents: "parent_library_X_test_not_applicable", sex = Y k-mers + role). KR sex = F by PI decision. COPY-AND-EDIT.
Phasing / concordance (PP/PIPELINE.sh): p01_phase_concordance.py <res> ×4 (hard-coded SRC, parent and ASM dicts, e.g. Apollo mother phase from Apollo_h1/h2 if genotype agreement ≥ 0.95), p03_bfreq_transmission.py, p05_validate_concordance.py, p04_export_viewer.sh; then ChrX sex-aware via hop_atlas/tools/atlas_layers.py xconc $STG (LR2/scripts/x11_sexaware_x_concordance.py). Crosses only. COPY-AND-EDIT (OSU fork o01_phase_osu.py).
Pedigree: atlas_layers.py pedigree $STG (GENERIC); name matching in LR2/scripts/p01_pedigree_build.py (SYN, UNVER, EXCL, CROSS); new cross = one CROSS line.
Dependency order (hop_atlas/components/DOSAGE.md §2–3): S3 per reference (<ref>_wgsmd) → S4f counts → clade calls from S5 (build_cn.py reads MCW/calls/<CALLSRC[pop]>_1Mb_calls.tsv.gz; ancestry-aware CN needs them) → v1 assets (/data/hop_data/hop_atlas/dosage_sources/2026-09-28/dosage_explorer/scripts/build_all.py, build_assets.py <ref>) → v2 builder /data/hop_data/dosage_explorer_v2/stage/scripts/rebuild_and_validate.sh (build_all_cn.sh → worked_example.py → fill_defs.py → make_page.py → validate_cn.py) → hop_atlas/tools/relocate_dosage.py.
- Hard-coded lists to edit: build_all_cn.sh (ref × pop loops), validate_cn.py and check_cn_browser.py (REFS, POPS), v1 build_all.py, mr_config.py (REFS), build_cn.py (CALLSRC, FRAME, REFHAP, EXCLUDE_BASE).
- WGS specifics: 100 kb bins; duplicates excluded; Apollo depth from ~2× subsample; X not readable as copy number in Cascade/Wye/Apollo (memory per-chromosome-dosage). Pan-X instrument = annevo_eval/methods/sexchrom_dosage/hop_xdosage (count --input <bam>, call --manifest), not in the served atlas.
- Generic package: annevo_eval/methods/autosomal_dosage/hop_adosage (cn --bins … --manifest … --assay …) — merged to main in commit 029bf81; DOSAGE.md still says "unmerged branch" (stale).
- KR and Carey: not in the dosage atlas.
T=hop_atlas/tools/atlas_layers.py # introgression layers (annevo_eval/scripts/atlas_layers.py is a 285-byte shim)
python3 $T stage --name <change>; STG=/data/hop_data/lineage_refinement/stages/<change>
python3 $T lineage $STG; python3 $T pedigree $STG; python3 $T xconc $STG
python3 $T serve $STG --port 8907; python3 $T test $STG --port 8907; python3 $T deploy $STG # deploy needs PI approval
D=hop_atlas/tools/atlas_deploy.py # dosage: stage dosage / build dosage / test dosage / deploy dosage [--dry-run]
Base-atlas JSON (S6) has no stage tool of its own; OSU used a hand-copied stage on :8901 and o09_swap_into_served.sh (md5-guarded). Rules: hop_atlas/README.md (never copy an old stage over the served copy; CHANGELOG entry; PI approval).
| Apollo | KR | Cascade parents | Wye parents | Carey 2026 | |
|---|---|---|---|---|---|
| Download | NCBI prefetch (box); 10 truncated → .bz2 repair |
ENA curl + md5 | ENA curl, gzip -t only |
ENA curl + md5 | SRA ODP on Lepton + SDL md5 + read_count |
| Read quirks | mother 35–265 bp; father 4 runs merged; 2 runs lack ENA read_count | GA-IIx, all-N R2 reads | 150 bp | 250 bp → pbrun crash untrimmed; one ref needed 150 bp crop | NovaSeq X / 6000 |
| Atlas BAMs | P/m21_wgsmd/bam/apollo/ (~2× subsample) |
local full depth | P/m21_wgsmd/bam/cascade/ (full) |
P/m21_wgsmd/bam/wye/ (full) |
AWS/PBSS carey2026/bam/21110M/ (full) |
| Route-B targets | 500 bp thinned (2.34 M) | full catalogue per chromosome | full catalogue | full catalogue | none (no base atlas) |
| EM params | own apollo_bam fit |
borrowed apollo_bam | shared with GBS progeny (cascade_bam) |
shared with GBS progeny (wye_bam) |
borrowed apollo_bam (lineage only) |
| Lineage layer | LIMITED_COH |
LIMITED_COH (ATLAS_COH kr→cordifolius) |
LIMITED_KEYS |
LIMITED_KEYS |
LIMITED_COH (copies in CAR/layers/) |
| Sex | MSY caller + Apollo masks, 1 Mb X + yb |
F (PI) | Y k-mers + role; Apollo masks | Apollo masks | not run |
| Phasing | progeny linkage + Apollo h1/h2 assembly | n/a | progeny linkage | progeny linkage | n/a (panel) |
| Dosage | yes (6 refs) | no (separate sync/2026-09-30/cordifolius_KR_multiref/) |
yes | yes | no |
Canonical table: annevo_eval/sample_identity/{sample_identity.tsv,identity_pairs.tsv,README.md}. Built by sync/2026-10-03/atlas_evidence_rebuild/scripts/e01_identity_table.py + e03_lineages.py (never hand-edit; re-run sync/2026-10-04/verified_sex_update/scripts/v01_identity_sex.py after). Same-plant groups (P####) = connected components of merged=True edges; status ok / duplicate / unresolved / non-progeny / different-plant / mislabel / mixture.
Naming: each library stays its own atlas row, keyed <cohort>|<sample> (WGS keys: run accession for Cascade parents and Carey; alias for Apollo/Wye/KR). Rows are never merged; identity is a group label, plus pedigree matching (p01_pedigree_build.py: exact / variant / synonym / approximate / unverified).
Identity checks used (records):
| Pair | Method | Result | Source |
|---|---|---|---|
| WGS cross parents vs panel GBS (Cascade, Apollo, WyeTarget) | 5 Mb clade concordance + opposite-hom count | 0.995–1.0, opp-hom 0 → merged | identity_pairs.tsv (h02, sync/2026-09-30/haplotype_incompatibility/tables/pedigree_identity_pairs.tsv) |
| Apollo_mother vs Apollo h1/h2 assembly | clade concordance; assembly provenance | same plant | same |
| OSU Cascade WGS (ERR11608298) vs OSU/USDA/divpanel Cascade GBS | cross-call-set KING-robust + IBS0 on shared records, both DP ≥ 8 (osu_population_qc/scripts/10_crossdata_identity.py, 14_crossdata_tables.py); rule same ≥ 0.354 & IBS0 ≤ 0.005 |
GBS-vs-WGS 0.231 "ambiguous (GBS-vs-WGS attenuation)"; WGS vs USDA 0.387 same | sync/2026-10-02/osu_population_qc/REPORT.md |
| same, within one joint call (GBS + deep) | plink2 KING at DP ≥ 15, MAF ≥ 0.05 | 0.387 / 0.420 | sync/2026-10-02/osu_21110M_genotyping/README.md:106 |
| transmitted-allele test | fraction of a sample's sites carrying alleles the Cascade mother transmitted | WGS 0.877 rank 1 | osu_population_qc/scripts/13_cascade_transmitted.py |
| lineage-layer agreement by assay pairing | same-plant pairs, SNP v5 vs tag, bins identical | tables only | LR2/scripts/x03_cross_assay.py → LR2/tables/X03_cross_assay_{pairs,summary}.tsv; x05_pair_crosstab.py cascade ERR11608298 osu cascade |
Recorded caveats: KING thresholds calibrated on GBS–GBS pairs; GBS-vs-deep pairs fall below them (REPORT calibration note). Cascade WGS mother vs OSU Cascade GBS is unresolved in sample_identity.tsv. Carey overlaps are "recorded, not resolved": Carey Cascade SRR37629444 is a different library from ERR11608298; Carey 19058M possibly = USDA 19058_M (DATA_CATALOG carey2026_wgs_reads). The OSU cross uses WGS ERR11608298 as mother in osu2022005_21110M, flagged SUPERSEDED (mother = WGS) in the explorer (sync/2026-10-02/osu_explorer_update/README.md:25); the served OSU atlas uses the OSU GBS cascade library as mother (TECH §1.3).
k08, OSU o07). Carey is blocked here.MCW/jobs/*.tsv, targets_apollo_500bp.tsv.gz).MCW/jobs/apollo_jobs.tsv → m21_wgsmd + PCD/scripts/00d_build_manifest.py (inferred). Full-depth BAMs exist under PBSS lepton_linear/apollo_21110M/ (inferred path from LL PUSH_PREFIX defaults).ADDING_DATA.md).s24 asserts 1,272 samples; only the OSU merge pattern exists. Not run for Carey.gzip -t only — record count vs ENA never recorded for them).sample_identity.tsv row kr:KR lists run_accession DRR024404–407 (the SW75 runs), not KR's DRR024452–456. DOSAGE.md says hop_adosage is unmerged (merged in 029bf81). ADDING_DATA.md/INTROGRESSION.md cite cluster/lepton_linear correctly but the task brief's cluster/lepton_linear does not exist. TECH §10.2 says the OSU sync scripts folder is empty (it is not).FAILED/absent in the 2026-10-08 19:07 QC table; whether a re-run is in flight is not recorded in the folder (check CAR/scripts/lepton/poll_carey.sh output / Lepton job state before acting).| Stage | Start from | Why |
|---|---|---|
| Download + QC | CAR/scripts/lepton/fastq_worker.sh + sra_stagein_worker.sh |
SDL md5 + exact read_count gate + ODP speed; ENA-form headers |
| Align | LL/linear_job.sh (config-only, resumable, dup gate, read-count anchor) |
already GENERIC; add markdup WGS defaults, per-ref loop from PCD/job_ref.sh |
| Per-sample bundle | CAR/scripts/lepton/carey_gpu_worker.sh |
one job does align ×2 refs, QC, mosdepth, L02 counts, kt tags, push |
| QC table | CAR/scripts/build_sample_table.py |
measured depth, decoded instrument, anchors |
| Counts | MC/scripts/m01_count_one.sh, LR1/scripts/l02_count_one.sh, SX/scripts/s10_pileup_one.sh, PCD/scripts/lepton/count_one_v2.sh |
already CLI-shaped; same mpileup family, differ only in target list / region |
| Calls | o03/o04 (OSU forks of m03/m04) |
show exactly which constants a new cohort needs |
| Export | OSU o07/o08/o17/o09 + TECH §10.2 |
the only complete add-a-cohort sequence |
| Layers/deploy | hop_atlas/tools/atlas_layers.py, atlas_deploy.py |
GENERIC; needs a cohort manifest in place of COH/LIMITED_COH |