Hop atlas home · Overview · Changelog · Architecture · Adding_Data · Introgression · Ribbon · Dosage · Phylogeny · Bundle · Ingest · Ingest_Gbs · Ingest_Wgs · Ingest_Grasdi · Ingest_Rnaseq · Ingest_Assembly

Rendered from hop_atlas/ingest/GRASDI.md (2026-10-13)

Ingest runbook: GRAS-Di cohorts → hop atlas web app (:8900)

Status: reconstructed from project history, 2026-10-08. Read-only reconstruction; nothing here was re-run. Only GRAS-Di cohort so far: Saaz ♀ × 10-12 (HL1012) ♂, DDBJ PRJDB19078. Every step below was done once, for Saaz, with paths and sample IDs hard-coded. No generic GRAS-Di pipeline exists. Labels used below: - GENERIC: runs for a new cohort with config or arguments only. - COPY-AND-EDIT: a dated, Saaz-specific script; copy it and change the listed items. - (inferred): not recorded; deduced from adjacent records. - MISSING: no record found.

Path abbreviations (same as annevo_eval/docs/INTROGRESSION_ATLAS_TECHNICAL.md):

abbrev	path
`CI` / `CIW`	`sync/2026-09-29/clade_ab_imputation` / `/data/hop_data/clade_ab_imputation`
`MC` / `MCW`	`sync/2026-09-30/multires_clade` / `/data/hop_data/multires_clade`
`UA`	`sync/2026-09-29/universal_introgression_atlas`
`PP`	`sync/2026-09-30/phased_parents_switch_concordance`
`SX`	`sync/2026-09-30/sex_chromosome_lineages`
`LR1` / `LR2` / `LRW`	`sync/2026-10-06/lineage_refinement` / `sync/2026-10-07/lineage_refinement_v2` / `/data/hop_data/lineage_refinement`
`PCD` / `MR`	`sync/2026-09-24/per_chromosome_dosage` / `sync/2026-09-25/per_chromosome_dosage_multiref`
`G`	`/data/hop_data/saaz_1012_grasdi`
`SA`	`sync_archive`

1. Scope

1.1 The cohort we hold

item	value	source
accession	DDBJ PRJDB19078 (257 runs, 514 FASTQ); paper Akagi et al. 2025 Nat Plants	`SA/2026-07-29/scripts/14_download_saaz_1012_graSdi.sh` header
samples	249 progeny + DRR613194 (Saaz ♀) + DRR613195 (10-12 ♂) + 6 unrelated cultivars DRR628685–690 (SW, Kazbek, R6KK1, R6K4, Agnus, R6M2)	`G/calls/cultivars.tsv`; `annevo_eval/data_inventory/catalog.yaml` id `PRJDB19078`
run vs sample	one run per sample; run accession = sample ID	`MR/scripts/00d_build_manifest.py:42`
read layout	paired-end 2×150 bp, HiSeq X (instrument E00582)	`cluster/lepton_zenith/config/saaz.env` (`PAIRED=1`)
depth	parents 19.0 M / 16.9 M reads; progeny ~4–6 M (later kt run: median 8.86 M reads per BAM)	`SA/2026-08-17/grasdi_alignment_21110M/README.md` §1; `LR2/scripts/lepton/PIPELINE_lepton.sh:68`
raw reads now	PBSS only. `s3://general-purpose/skothenhill/archive/saaz_1012_grasdi/` (514 objects). Local `G/fastq/` is gone.	`archive_ledger.json` entry `saaz_1012_grasdi` (`local_present: false`)
BAMs (21110M)	PBSS `…/archive/saazx1012_align_21110M/data/hop_data/saaz_1012_grasdi/align_21110M/<DRR>.bam(.bai)` (786 objects, 170 GB); local `G/align_21110M/` is gone	ledger entry `saazx1012_align_21110M`
other non-progeny	DRR613305/306/307/388/435/436 are excluded as "not Saaz × HL1012 progeny" (role `non-progeny`)	`annevo_eval/sample_identity/README.md`; memory `clade-ab-imputation`

Stale catalogue text: catalog.yaml still says the BAMs are "regenerable from /data/hop_data/saaz_1012_grasdi/fastq (retained)". That directory no longer exists; restore from PBSS instead.

1.2 How GRAS-Di differs from GBS (what changes downstream)

property	GBS (ApeKI / PstI)	GRAS-Di	consequence / source
locus definition	restriction cut site	random-primer PCR amplicons; no enzyme; read starts are primer sites (top start-5mer only 5.2 %)	No cut-site model, no digest-based callability. GBS tag-space methods that assume "the 5′ 64 bp is the locus" do not apply. `ASSAYS.md` §1.5 table, §"2026-09-05 graph-painting callability"
primer	n/a	R1 tags dominated by `GCACCTT[CT]TC[CT]A` and variants (~49k top tags); non-random composition to ~position 20	40 bp R1 tags place at only 0.4 % with minimap2; use exact k-mer lookup (`kmc -k40`) if tags must be placed. `ASSAYS.md` ~l.1774, ~l.1804
insert size	~199 bp	median 62 bp (alignment-free), mode ~45–50 bp; TLEN (131 bp) is censored and overstates it	`SA/2026-08-17/grasdi_alignment_21110M/README.md` §4
adapter	—	Nextera read-through on 59 % of reads	same README §3
mapping rate (21110M, R1)	98 %	66–68 %; ~47 % of reads at MAPQ ≥ 20. Hard ceiling, not fixable by trimming	same README §2–3
genome coverage per sample	—	0.03–0.10 % breadth; ~84 sites/Mb; depth/covered 21–47× vs depth/genome 0.008–0.046×	same README §1–2
marker type	codominant SNP at cut site	~53 % dominant (primer-3′ variant kills the amplicon) / ~47 % codominant	`ASSAYS.md` §1b.2
library skew	—	very top-heavy (712 tags = 31.8 % of reads); use rank-matched comparisons	`ASSAYS.md` §1b.4
duplicate rule	`--no-markdups`; mpileup without DUP	same as GBS: `--no-markdups`; `--ff UNMAP,SECONDARY,QCFAIL`; depth `samtools view -F 0xB04` (DUP kept). Gate `samtools view -c -f 1024 == 0`	`ASSAYS.md` §1.1, entry "2026-09-25 --no-markdups is for GBS/GRAS-Di ONLY"
joint-call sparsity	—	median genotype DP 0; 11.3 % genotypes DP ≥ 2. Expected; not a duplicate problem	`ASSAYS.md` §1.3
genotype depth floor	Havill 6	10, measured from testcross het fraction (~16 % dropout remains at DP ≥ 10)	`SA/2026-08-18/saazx1012_linkage_pipeline/README.md`; `G/calls/measure_mindp.sh`
lineage layer	European lineages shown	assay-limited: European lineage hidden (cohort in `LIMITED_COH`)	`LR2/scripts/e03_viewer_export.py:18-30`; `ASSAYS.md` "Group allele-frequency models…"

2. Stages

Order: download → QC → alignment → calling → NA/EU base calls → layers (phasing, sex, subspecies/lineage, pedigree) → dosage → stage / test / deploy.

Stage 0. Manifest and roles (do this first)

Every downstream script hard-codes the parents, truth plants and exclusions (§3 G1). Before anything else, write down: - runs → samples; - mother and father IDs; - non-progeny and control samples; - truth plants (only if the parents have assemblies in the 13-haplotype panel); - triploid flags.

There is no manifest file or generator. Saaz's roles live inside the scripts listed in Stages 5–8.

Stage 1. Download → PBSS


scripts	`SA/2026-07-29/scripts/14_download_saaz_1012_graSdi.sh` (main); `/data/hop_data/resume_prjdb19078.sh` (resume: retries files whose size ≠ ENA size, MD5 check)
input	ENA file report `/data/hop_data/PRJDB19078_files.tsv`, columns `run_accession sample_title fastq_ftp fastq_md5 fastq_bytes`
tool	`curl -s -L --retry 3` from `ftp.sra.ebi.ac.uk` over https; `md5sum` vs `fastq_md5`; one retry; `aws --endpoint-url https://pbss.s8k.io s3 cp` to PBSS
output	`s3://general-purpose/skothenhill/archive/saaz_1012_grasdi/<run>_{1,2}.fastq.gz` (local copy kept at the time, since removed); log `/data/hop_data/PRJDB19078_download.log`
QC gate	MD5 = ENA md5 for every mate. Object count in PBSS = 2 × runs. Saaz PBSS was missing `DRR613372_2.fastq.gz` (513/514), found only on 2026-09-04 by diffing listings (`catalog.yaml`, zenith_saaz_v2 provenance)
gotchas	The file report's request URL is MISSING (inferred: ENA `filereport?accession=PRJDB<…>&result=read_run&fields=run_accession,sample_title,fastq_ftp,fastq_md5,fastq_bytes`). The accession was not in the paper's data statement; it was found by enumerating the DDBJ project series (script header). CLAUDE.md: new transfers should use `aws s3 sync`, not per-file `cp`.
register	`catalog.yaml` entry + `annevo_eval/data_inventory/refresh.sh`; archive ledger entry (`archive.sh`)
class	COPY-AND-EDIT. Edit: manifest path, PBSS prefix, local dir.

Stage 2. Read QC


what was done (linear route)	No read prep. Raw DDBJ FASTQ went straight into `pbrun fq2bam`.
trimming pilot	`SA/2026-08-17/grasdi_alignment_21110M/scripts/run_trim_pilot.sh` (output in `G/trim_pilot/`). Ran `fastp` with explicit Nextera adapters (`--adapter_sequence CTGTCTCTTATACACATCTCCGAGCCCACGAGAC --adapter_sequence_r2 CTGTCTCTTATACACATCTGACGCTGCCGACGA --detect_adapter_for_pe --length_required 25`). Mapping 66.04 % → 65.77 % (−0.27 pt) and 2.4 % of reads lost. Trimming was rejected for alignment.
graph route (not used by the atlas)	`cluster/lepton_zenith/config/saaz.env` did trim: `FASTP_EXTRA=<Nextera adapters>`, `FASTP_MIN_LEN=25`.
QC measurements available	`SA/2026-08-17/grasdi_alignment_21110M/scripts/{coverage_stats.py, insert_sizes.py, coverage_coord_compare.py, depth_histograms.py}` (10-sample probe). Reference-free containment QC in `G/amplicon_world/` (`run_amplicon_world.sh`; `RESULTS.md` "QC outliers"): rank-matched top-5k containment ≈ 99.6 % for progeny vs ~47 % for unrelated cultivars.
QC gate	None formalised. (inferred) Flag samples with low containment or low library skew. DRR613305–307 were flagged this way (`G/amplicon_world/RESULTS.md` l.146–152) and later became non-progeny.
class	COPY-AND-EDIT (the probe scripts). There is no standalone GRAS-Di QC tool.

Two decisions conflict in the records: the linear route used no trimming, the graph route used Nextera trimming. Choose one per cohort and record which.

Stage 3. Alignment

Reference used by every atlas layer: 21110M linear, /data/hop_data/ref_21110M_male/male_ref.fa (Chr01–09, ChrX, ChrY; Akagi numbering; bwa index present). The atlas bins, diagnostic catalogue and lineage catalogue are all on this frame (INTROGRESSION_ATLAS_TECHNICAL.md §1.1).

Graph vs linear: - The v5 13-haplotype graph route (Zenith; cluster/lepton_zenith/config/saaz.env; PBSS lepton_zenith/saaz_v2/) was run but is not used by any atlas. - On that route Saaz lost 16× sites and 33× markers vs linear, and the graph has no ChrY (memory saaz-pangenome-vs-linear). - The parental-4-frame re-surjection (cluster/lepton_saaz_par4/) and the G/giraffe_v5restricted/ attempt are also non-atlas. - Use linear 21110M for atlas ingest.

3a. As actually done for Saaz (local GPU, Aug 2026)


scripts	`SA/2026-08-18/saazx1012_linkage_pipeline/scripts/align_all.sh` (all samples, resumable, stops below 60 GB free); `align_parents.sh`; `SA/2026-08-17/grasdi_alignment_21110M/scripts/align_grasdi_21110M.sh` (8-progeny probe)
tool + flags	`docker run --gpus all nvcr.io/nvidia/clara/clara-parabricks:4.7.1-1 pbrun fq2bam --ref male_ref.fa --in-fq <R1> <R2> --read-group-sm <DRR> --read-group-id-prefix <DRR> --no-markdups --out-bam <DRR>.bam`
output	`G/align_21110M/<DRR>.bam` (257 BAMs, 159 GB), later archived to PBSS `saazx1012_align_21110M` and deleted locally
QC gate	"quickcheck clean, 0 failures" (`SA/2026-08-18/…/README.md`). (inferred) the gate was `samtools quickcheck`.
class	COPY-AND-EDIT (FASTQ dir, ref, out dir). Superseded by 3b for new cohorts.

3b. Generic route for a new cohort: cluster/lepton_linear/


files	`README.md`, `linear_job.sh`, `submit_linear.py`, `config/<pop>_<ref>.env`, sheets `sheets/`
template	`config/havill_clean_21110M.env` (parents and progeny on the same reduced-representation assay)
GRAS-Di settings	`PAIRED=1`; `MARKDUP=0`; `PARENT_MARKDUP=0` (the driver defaults it to 1 = WGS parents, `linear_job.sh:178`); `CALL_FF=UNMAP,SECONDARY,QCFAIL`; `GBS_BATCH=24` (pooled pbrun is always `--no-markdups`); fastp: either `DO_FASTP=0` (Saaz linear precedent) or `FASTP_EXTRA=<Nextera adapters> FASTP_MIN_LEN=25` (Saaz graph precedent)
built-in gate	standing duplicate check `samtools view -c -f 1024` per BAM vs expected state (`linear_job.sh:1052-1100`)
status for GRAS-Di	Never run on a GRAS-Di cohort (no `saaz.env` in `cluster/lepton_linear/config/`). Settings above are (inferred)* from the Saaz records + the driver's knobs.
class	GENERIC. Write one `.env` file.

3c. Other Saaz BAM sets that exist (know which one a downstream step reads) - MR per-reference re-alignment (Lepton, pbrun 4.7.0-1, pooled --in-fq-list, --no-markdups, unique LB:<sample>), on 6 references (m21, apollo_h1/h2, saaz_h1, hl1012_h1, cascade). - PBSS: s3://…/archive/per_chromosome_dosage_multiref/<ref>/bam/saaz/. - The m21/bam/saaz set is what the sex lane read. - Record: MR/PIPELINE.sh, MR/scripts/lepton/job_ref.sh. - HL1012 father-frame arm (sync/2026-09-11/saaz_parental_refs/PIPELINE.sh). - pbrun fq2bam --no-markdups --num-gpus 1 on hl1012_h1.oriented.fa. - Per-BAM gates: samtools view -c -f 1024 == 0 and records ≥ 0.97 × (pairs × 2). - Mapped-only CRAMs on /data2. Not used by the atlas.

Gotchas: - The ~34 % unmapped fraction is assay physics (short inserts); do not chase it (SA/2026-08-17/…/README.md §3). - Never run duplicate marking. ASSAYS.md §1.1. - The 9 graph-route Saaz samples that map < 50 % (memory saaz-pangenome-vs-linear) are a graph-route fact; it is unrecorded whether the same samples are low on linear.

Stage 4. Variant calling (cohort VCF; route-V input)


script	`G/calls/call_saazx1012.sh` (copy: `SA/2026-08-18/saazx1012_linkage_pipeline/scripts/call_saazx1012.sh`)
tool + flags	bcftools 1.21 (`/opt/conda/envs/variant/bin/bcftools`): `mpileup --fasta-ref male_ref.fa --bam-list bamlist.txt --regions <chr> --min-MQ 20 --min-BQ 20 --ff UNMAP,SECONDARY,QCFAIL --annotate FORMAT/AD,FORMAT/DP` \| `call --multiallelic-caller --variants-only --format-fields GQ`; per chromosome, 4 in parallel; `concat` → `.vcf.gz` + `.tbi`
input	`G/calls/bamlist.txt` (257 BAM paths)
output	`G/calls/saazx1012_21110M_genomewide.vcf.gz` (7,148,462 sites × 257). Also `…_genomewide.bcf` (2026-09-17, producer MISSING) and `…_progeny.vcf.gz` (251 samples = minus the 6 cultivars in `G/calls/exclude_samples.txt`; the subset command is MISSING, only `subset.log` "SUBSET_DONE"; (inferred) `bcftools view -S ^exclude_samples.txt`).
QC gate	`G/calls/measure_mindp.sh` (testcross het fraction by progeny DP, both arms) → floor 10. `G/calls/retention.log`.
gotchas	`--ff` without `DUP` is mandatory. It also admits SUPPLEMENTARY records, which can inflate ALT AD at a focal locus (`ASSAYS.md` "mpileup --ff … still admits supplementary"). mpileup default `-d 250` applied (implicit). The atlas reads the `_genomewide` file (all 257 samples, cultivars included), not `_progeny`.
class	COPY-AND-EDIT. Edit: `REF`, `O`, bamlist, chromosome list. The `lepton_linear` job does its own calling (`UNTARGETED=1`, `MIN_MQ_SET="0 20"`, `MAX_DEPTH=500`) if 3b is used.

Stage 5. NA/EU (clade A/B) base calls into atlas bins

Saaz entered by route V (VCF AD). Route B (BAM pileup at catalogue sites) was not run for Saaz's base calls.

Gotcha (route V): the cohort VCF is variants-only, and REF = clade A (EU) at every autosomal catalogue site. Saaz calls are measurably biased: - 1,401 progeny bins AB→AA at 5 Mb when recounted on all sites; - Chr06 AB 5.8 % → 0.01 %; - dB 0.45 → 0.56.

Source: sync/2026-10-03/variants_only_audit/README.md l.27–56; memory variants-only-site-selection-trap. The audit lists Saaz as still VCF-based / AFFECTED in the served atlas.

5V. Route V, as done for Saaz

step	script	Saaz-specific edits
1. AD at diagnostic sites	`CI/scripts/07_extract_cohort.py saaz G/calls/saazx1012_21110M_genomewide.vcf.gz` → `CIW/cohorts/saaz.npz` (`bcftools view -T CIW/diag/targets_all.tsv.gz -m2 -M2 -v snps \\| query … [%AD]`; 150 bp grid thinning; no QUAL/DP/GQ/GT filter)	cohort key + VCF path on the command line (`CI/PIPELINE.sh` l.~45)
2. EM fit (5 Mb, 10 iterations, no convergence test)	`CI/scripts/08_impute.py saaz` → `CIW/imputation/saaz_params.json`	`TRUTH["saaz"]` (`08:53-54`); `TRIP` (none for Saaz)
3. evaluate / τ / depleted-bin rule	`CI/scripts/09_evaluate.py`	`PARENTS["saaz"]` (`09:88`), `EXCL["saaz"]` (`09:165`, the 6 cultivars), `PLANTHAPS`
4. multires calls (500 kb / 1 / 2 / 5 Mb; params held fixed)	`MC/scripts/m04_impute_multires.py saaz` → `MCW/calls/saaz_<res>_calls.tsv.gz`	`TRUTH` (`m04:51`), `CROSS_VCF` (`m04:144`)
5. served call-set choice + reliability	`MC/scripts/m05_evaluate.py` → `MC/tables/Q0_callsets_used.json`	`FINAL`, `PARENTS`, `NONPROG`, `HAPS` (`m05:22-34,64`)
6. transitions, hot spots	`m06_transitions.py`, `m06b_flip_hotspots.py`	`PARENTS`, `NONPROG`
7. viewer export	`m07_viewer_export.py` (+ `m07b_check_viewer.py`)	`PARENTS`, display labels, `NONPROG` (`m07:32-37,149`)
8. universal-atlas pieces	`UA/scripts/u03_support.py` (roles/labels, `SAAZ_NONPROGENY`), `u06_viewer_data.py`, `u13_similarity_order.py`, `u14_transmission.py`, `u17_multires_detail.py saaz` (→ `detail_mr/`)	cohort lists `COH`, `PAR`, groups

Class: COPY-AND-EDIT. Saaz was added in the original multi-cohort build (09-29/09-30), not by a patch.
For a new cohort, follow the OSU patch path instead (technical doc §10.2): o00 labels → counts → o03 npz → o04 (copy of m04 with TRUTH/TRIP lines) → o05 evaluate → stage the viewer [manual] → o07 export + o08 preset → o17 detail → tests → o09 md5-guarded swap.
Scripts: /data/hop_data/osu_introgression_atlas/scripts/.
For route V, use the wyerna variant: 07_extract_cohort.py <c> <vcf> → 08_impute.py <c> → o04.

5B. Route B (BAM pileup): recommended by the audit, never run for GRAS-Di base calls (inferred) - Count: MC/scripts/m01_count_one.sh with the GBS branch (--ff UNMAP,SECONDARY,QCFAIL, --min-MQ 20 --min-BQ 20 -a FORMAT/AD -T targets_all.tsv.gz, gate > 1000 lines). OSU Lepton variant: count_osu_aims.sh. - Then m03_build_bam_cohort.py (≥ 5 samples, 150 bp grid) → o04 <c>_bam. - The m01_count_one.sh type switch is GBS/WGS: give GRAS-Di samples type GBS so duplicates stay in. (inferred) - Job tables have no generator (technical doc §1.2).

Truth plants and accuracy. Saaz is the only cohort whose parents are both panel assemblies (Saaz/Saaz_h2, HL1012/HL1012_h2), so it has truth plants. - Measured truth accuracy: 0.9495 (500 kb) to 0.9741 (5 Mb). - Served with viewer warnings at every resolution (technical doc §4.6). - A new GRAS-Di cohort without assembled parents gets no truth plant, and reliability rests on callability and parent consistency only.

Stage 6a. Phasing and a/b/c concordance (crosses)


scripts	`PP/scripts/p01_phase_concordance.py <res>` × 4 → `p03_bfreq_transmission.py` → `p02_summaries.py` → `p05_validate_concordance.py` → `p04_export_viewer.sh` (copies into served `phase/`, pre-dates staging) → `run_tests.sh <res>`
inputs	`MCW/calls/saaz_<res>_calls.tsv.gz`; `MCW/stage1/hap_bin_<res>.tsv` (assembly phase for Saaz/HL1012); served `multires/atlas_samples_<res>.json` (roles, exclusions)
Saaz edits	`SRC`, `PAR`, display names, `ASM` (`p01:40-46`); `EXCL` is read from the 5 Mb transmission JSON (`p01:55`)
after	`python3 hop_atlas/tools/atlas_layers.py xconc $STG` (sex-aware ChrX a/b/c; GENERIC)
QC gate	Assembly phase = progeny phase at linked junctions (HL1012 reproduced); random-flip control ~0.5; non-progeny should score (c) (Saaz 41/54) (memory `clade-ab-imputation`)
gotchas	Role inconsistency issue I-28: DRR628685–690 are role `progeny` for phasing but excluded by SX; 6 vs 12 exclusions differ by script. For a new cohort, use the OSU fork `o01_phase_osu.py` + `o10`.
class	COPY-AND-EDIT

Stage 6b. Sex chromosomes (ChrX / ChrY lanes)


sex per plant	`PCD` MSY caller (`msy_log10`, `xy_like`) from 1 kb read-start counts: `samtools view -q 20 -F 0xB04` (DUP kept) via `PCD/scripts/02b_run_manifest_1k.sh` (Saaz streamed from PBSS: `aws s3 cp - \\| samtools view -`). Consumed by `SX/scripts/s14_sex_table.py` (parents hard-coded F/M, `s14` PARENTS)
ChrY / ChrX-end pileups	`SX/scripts/s10_run_jobs.sh $W/jobs/s10_jobs.tsv 8` → `s10_pileup_one.sh`: `bcftools mpileup -f male_ref.fa -T targets_sex.tsv -r ChrY,ChrX:225000001-268239420 --min-MQ 20 --min-BQ 20 --ff UNMAP,SECONDARY,QCFAIL -a FORMAT/AD` on presigned PBSS URLs. Saaz BAMs = `MR` m21 set (`…/per_chromosome_dosage_multiref/m21/bam/saaz/`, 257 rows, type GBS). `s10_jobs.tsv` has no generator script (MISSING)
X caller	`s13_build_counts.py` → `s15_y_lineage.py` → `s16_x_caller.py --no-mask` → `s16b_hxr_male_mask.py` → `s16_x_caller.py` (Saaz X evidence = `CIW/cohorts/saaz.npz`, i.e. route-V VCF AD; `SRC`/`PARF` dicts `s16:32-34`) → `s17`–`s22` → `s24_viewer_export.py` → `s27_viewer_integrate.py`
OSU-era fork	`o11_sexchrom_osu.py` + Lepton `count_osu_sex.sh` (technical doc §10.2 step 10)
gotchas	21110M Y ≠ the cross father's Y lineage. Run the X:A depth check before reading chrX/chrY counts (memory `saaz-hl1012-arm-callset`). The graph route has no Y.
class	COPY-AND-EDIT

Stage 7. Subspecies / lineage layers (haplotype-resolution)

Procedure: hop_atlas/components/INTROGRESSION.md §4.3. Saaz steps:

step	script / command	Saaz specifics
BAMs to fs1	Lepton job restores from PBSS `saazx1012_align_21110M` → `/mnt/fs1/skothenhill/linear/saaz_1012_21110M/bam/` (size check + `samtools quickcheck`)	`LR1/scripts/lepton/PIPELINE_lepton.sh` §2
task list	`LR1/scripts/lepton/make_tasks.py` → `LRW/lepton_work/inputs/tasks_saaz.tsv`	hard-coded `saaz_samples.txt`, PBSS listing, `assert len == 257`
SNP-layer counts (L02)	`submit_count_l02.py --tasks tasks_saaz.tsv --workers 8 --par 4` → worker `count_l02.sh`: `bcftools mpileup -f male_ref.fa -T LRW/catalogue/targets.tsv.gz --min-MQ 20 --min-BQ 20 --ff UNMAP,SECONDARY,QCFAIL -a FORMAT/AD` (DUP kept for GRAS-Di)	AWS `bionemofw` transfer tier → `pull_counts.sh saaz` → `LRW/bamcounts/saaz/`; spot-check `local_spotcheck.sh` (DRR613431 md5-identical)
sparse matrix	`LR1/scripts/l03_build_sparse.py saaz` (`l03_run_all.sh`)	keep `len(REF)==1`, drop duplicate (CHROM,POS) (`ASSAYS.md`)
SNP calls v6	`LR2/scripts/c01_call_levels_v6.py saaz --groups C00_groups.tsv --tag _v6` (`c01_run_all_v6.sh`)	dicts `CALLSET`, `PARAMS` (→ `CIW/saaz`), `ASM_PLANT` (`c01_v6:53-58`)
tag extraction	Lepton `submit_kt.py --tasks tasks_saaz.tsv`: `samtools view -F 0x900 <bam> \\| kt --mode scan --out <DRR>.kt --min-count 2` (GRAS-Di = `scan`, not `gbs5scan`; `-F 0x900` like GBS) → `pull_kt.sh saaz` → `LRW/tags/saaz/`	`LR2/scripts/lepton/PIPELINE_lepton.sh` l.40, 50, 57–60 (local `cmp` check)
tag sparse + calls	`LR2/scripts/t03_run.sh saaz`; `EXTRA='--minref 1' c01t_run.sh _t4 <groups> saaz` with `--l0-from calls/saaz_v6`	—
register in export	add to `COH` in `e03_viewer_export.py:28`, `e04_detail_export.py:18`, `viewer/l23_lineage_v3_test.py:25`; add to `LIMITED_COH` (`e03:30`) and `LIM_COH` (`l23:26`)	GRAS-Di: European lineage hidden
export / test / deploy	`atlas_layers.py stage` → `lineage $STG` → `test $STG --port 8907` → `deploy $STG`	GENERIC

Gotcha: the ASSAYS rule "a site counts only if every competing group has reference data" (c01 v5+) gives GRAS-Di fewer calls, not wrong ones.
Class: COPY-AND-EDIT up to export; GENERIC from atlas_layers.py lineage on.

Stage 7b. Pedigree link

Add one CROSS entry in LR2/scripts/p01_pedigree_build.py:92-93. Saaz: 'saaz': ('Saazer', 'HL1012', 'Saaz x HL1012 cross').
Add a SYN entry if the parent name differs from the pedigree (saazer: ['saaz']).
Then run atlas_layers.py pedigree $STG (GENERIC).
Review tables/P01_pedigree_matches.tsv.

Stage 8. Dosage atlas

step	record	Saaz specifics
1. 21110M read-start counts + AB	`PCD/PIPELINE.sh`: manifest `/data/hop_data/per_chromosome_dosage/manifests/saaz.tsv` (built by hand, MISSING generator); counts `samtools view -q 20 -F 0xB04` per 1 kb (`01b_count_bins_1k.sh`); AB from the route-V VCF: `bcftools query -i 'TYPE="snp" && N_ALT=1 && QUAL>=20' -f '%CHROM\t%POS[\t%AD{0}\t%AD{1}]\n'` \| `03_allele_balance.py` (`03b_run_ab.sh saaz`, parents DRR613194/DRR613195; f at DP ≥ 8/20) → `04_depth_dosage.py`, `06_combine_flag_plot.py`	GRAS-Di treated as GBS (DUP kept)
2. multi-reference align + count	`MR/PIPELINE.sh`: `00d_build_manifest.py` (saaz block l.11, 42–43; assay `GBS` covers GRAS-Di) → `lepton/make_bundle.sh` → `lepton/submit_pcd.py <ref> --bundle main` × 6 refs; in `job_ref.sh`: pooled `pbrun fq2bam --no-markdups`, `count_one.sh`, 5 Mb-window `mpileup -q 20 -Q 20 --ff UNMAP,SECONDARY,QCFAIL -d 500 \\| call -m -v` → `ab_hist.py`; pull `counts`/`ab` back; `05_mr_ab_summary.py <ref> saaz`, `04_mr_depth.py`, `06_mr_combine_flags.py`	parents hard-coded `job_ref.sh:344,367`; cohort lists `job_ref.sh:150-154,390-421`
3. v1 assets	`/data/hop_data/hop_atlas/dosage_sources/2026-09-28/dosage_explorer/scripts/build_all.py`	ref list l.10
4. v2 CN builder	`/data/hop_data/dosage_explorer_v2/stage/scripts/`: `build_all_cn.sh` (`r × p` loop l.5), `build_cn.py` (`CALLSRC['saaz']='saaz'`, `FRAME`, `REFHAP`; l.323 labels group `'GRAS-Di'`, 1 kb bins), `validate_cn.py` / `check_cn_browser.py` (`REFS`, `POPS`)	clade calls from `MCW/calls/saaz_*` (route V)
5. relocate / test / deploy	`atlas_deploy.py stage dosage` → `build dosage $STG` → `test dosage $STG` → `deploy dosage $STG --dry-run` → `deploy` → `hop_atlas/tools/check_dosage_http.py`	GENERIC

Gotchas:
WGS-only duplicate rules (MARKDUP_WGS, -F 0xF04) must not touch GRAS-Di (ASSAYS.md 2026-09-25).
Allele-balance metrics fail as ploidy evidence on GRAS-Di. They track library depth and plate, and calibrated diploid parents score "triploid". Use the testcross-site het fraction instead (memory saaz-triploid-screen-result; ASSAYS.md §1b.5–1b.6).
Class: COPY-AND-EDIT for steps 1–4 (lists in ~5 files: hop_atlas/components/DOSAGE.md §2). GENERIC for step 5.
hop_adosage (generic) exists only on branch dosage-methods.

Stage 9. Stage → test → deploy (all atlases)

Introgression layers: hop_atlas/tools/atlas_layers.py stage|serve|test|deploy|lineage|xconc|pedigree|patch (components/INTROGRESSION.md §2).
Base-atlas cohort patch (OSU precedent):
Copy SV → OA/stage_viewer/ [manual].
Run o07/o08/o17.
Run run_tests.sh <res> ×4 plus a cohort-specific must-fail test (o21/o22/k21 pattern).
Swap with o09_swap_into_served.sh.
o07 resets index.html, so re-run o08 and then atlas_layers.py patch $STG (lineage + pedigree hooks).
Dosage: atlas_deploy.py … dosage.
Gate: viewer tests only when the PI asks (hop_atlas/README.md rule 3). Served changes need PI approval, recorded in the sync entry.
Register: register_product.sh for counts, labels and calls; catalog.yaml + refresh.sh.
Class: GENERIC (tools), except the base-atlas patch scripts.

3. Missing, ambiguous or inconsistent (explicit)

#	item	status
G1	Cohort manifest (runs, roles, parents, truth, exclusions, assay)	MISSING. Saaz roles are hard-coded in ≥ 15 scripts (`08`, `09`, `m04`–`m07`, `u03`/`u06`/`u13`/`u14`, `p01`, `s14`/`s16`, `c01_v6`, `e03`/`e04`/`l23`, `p01_pedigree_build`, `job_ref.sh`, `build_cn.py`)
G2	ENA file-report query that produced `PRJDB19078_files.tsv`	MISSING (column set known; URL inferred)
G3	Read-QC gate for GRAS-Di	MISSING. Only probes and reference-free containment QC exist. Linear (no trim) vs graph (Nextera trim) disagree
G4	Alignment QC gate actually applied to the 257 linear BAMs	"quickcheck clean" stated only. The `samtools view -c -f 1024` gate is not recorded for this set (inferred to be 0 from `--no-markdups` + `ASSAYS.md` §1.3)
G5	`_progeny.vcf.gz` subset command; `_genomewide.bcf` (09-17) producer	MISSING
G6	GRAS-Di through `cluster/lepton_linear`	never done; settings in 3b are inferred
G7	Route-B base calls for GRAS-Di	never done. Served Saaz calls remain variants-only VCF-based (AFFECTED per `sync/2026-10-03/variants_only_audit/`)
G8	Exclusion sets	RESOLVED 2026-10-08 (code + stage; not deployed): one manifest list of all 12, read by every recurrent script via `pipeline/exclusions.py` — see `SAAZ_EXCLUSIONS.md`. Was inconsistent (technical doc issue I-28): `09_evaluate` EXCL = DRR628685–690; MC/UA `NONPROG` = DRR613305/306/307/388/435/436; SX excludes all 12; phasing treats DRR628685–690 as progeny. Where the 613xxx "non-progeny" label was first assigned is not traced beyond `G/amplicon_world/RESULTS.md` (DRR613305–307 QC flag) and `sync/2026-09-24/coverage_atlas/INTERPRETATION.md`
G9	`SX` job table `s10_jobs.tsv`; `PCD` manifests; Route-B job tables	no generator scripts
G10	Which BAM set each layer read	Three exist: Aug `saazx1012_align_21110M` (lineage, PCD counts, base VCF); `MR` pooled `m21/bam/saaz` (sex lane, multiref dosage); HL1012 arm CRAMs (none). Not stated in any one place before this file
G11	EM convergence for Saaz	Not converged at iteration 10 (issue I-19). No refit rule when samples are added
G12	Per-sample mapping-rate gate on linear	not recorded
G13	Second GRAS-Di cohort	none held. Every "GRAS-Di rule" above is from n = 1 cohort

4. Scripts worth generalising first (from this reconstruction)

Cohort manifest + loader, read by all of the files in G1.
07_extract_cohort.py / m01_count_one.sh. Add an explicit assay argument (GBS | GRAS-Di | WGS) that sets the --ff DUP rule, so GRAS-Di is never silently classed WGS. Prefer route B.
cluster/lepton_linear GRAS-Di preset. config/_grasdi_template.env with PARENT_MARKDUP=0 and the fastp choice explicit.
Download: generalise 14_download_…sh + resume_prjdb19078.sh into one manifest-driven aws s3 sync-based fetcher with an object-count gate.
LR1/scripts/lepton/make_tasks.py. Drop the hard-coded assert == 257 and the per-cohort PBSS paths.
Dosage lists (DOSAGE.md §2) → manifest.