Hop atlas home · Overview · Changelog · Architecture · Adding_Data · Introgression · Ribbon · Dosage · Phylogeny · Bundle · Ingest · Ingest_Gbs · Ingest_Wgs · Ingest_Grasdi · Ingest_Rnaseq · Ingest_Assembly

Rendered from hop_atlas/ADDING_DATA.md (2026-10-13)

Adding data to the hop atlas web app: what is documented, by data type

Audit date: 2026-10-08; updated 2026-10-13 (Carey 2026 live, Zhang 2017 opt-in, cohort manifests, per-reference calls). Every cell below cites the file it rests on. Step-by-step runbooks per data type: ingest/README.md. Answer: no atlas has a documented, generic workflow for adding new samples, for any data type. - Generic and runnable (DS): alignment, then stage → build → test → deploy (tools/atlas_deploy.py, tools/atlas_layers.py). - Copy-and-edit only (REC): everything between, which exists only as dated, cohort-specific records.

Code Meaning
DS documented and scripted (generic, runnable)
REC record only: a dated, cohort-specific script set that could be copied and edited
NP not possible by design
ND not done (no record)
n/a not applicable to that atlas

Data type × atlas

Data type Introgression (base NA/EU calls) Introgression layers (subspecies / lineage · sex lane · phasing) Ribbon Dosage
GBS ApeKI, new cohort REC: route B (BAM pileup); OSU sync/2026-10-02/osu_introgression_atlas/scripts/o00–o18 + PIPELINE.sh; technical doc §10.2 Lineage REC (components/INTROGRESSION.md §4.3; sync/2026-10-07/lineage_refinement_v2/PIPELINE.sh); export, test and deploy DS, but cohort list COH is hard-coded in e03 / e04 / l23. Sex REC (o11_sexchrom_osu.py, sync/2026-09-30/sex_chromosome_lineages/). Phasing REC (o01_phase_osu.py, sync/2026-09-30/phased_parents_switch_concordance/), then atlas_layers.py xconc DS n/a REC: sync/2026-09-25/per_chromosome_dosage_multiref/PIPELINE.sh + about 5 hard-coded lists (components/DOSAGE.md §2–3)
GBS PstI REC: only within Cascade (92 PstI + 88 ApeKI progeny; annevo_eval/docs/ASSAYS.md §1.5); served via the cascade_bam route (sync/2026-09-30/multires_clade/) Lineage REC (Cascade in COH; tag extraction kt --mode gbs5scan) n/a REC (Cascade in the 09-25 record)
WGS short-read REC: Apollo via route B (multires_clade); KR via route K (KR/scripts/k0*). Carey 2026 (22 WGS): live since 2026-10-09 through its manifest + numbered stage scripts (cohorts/carey2026/, pipeline/c26_01…c26_25, c26_run_stage.sh; record /data/hop_data/cohorts/carey2026/HANDOFF_INTROGRESSION.md) Lineage REC: sync/2026-10-08/carey2026_hop_wgs/PIPELINE.sh + layers/ (WGS in LIMITED_COH); Carey shown since 2026-10-09, incl. subspecies evidence and the cordifolius level. WGS-variants level (Lw) REC: sync/2026-10-10/wgs_subspecies_variants/ (joint call of 30 WGS plants). Sex REC (Apollo; Carey c26_10, c26_20–c26_22: MSY reads + pan-X k-mer dosage must agree). Phasing REC (Apollo) n/a REC (Apollo; WGS with duplicate marking, 09-25)
GRAS-Di REC: Saaz via route V (VCF AD; /data/hop_data/saaz_1012_grasdi/calls/) Subspecies only (Saaz in LIMITED_COH; European lineages don't transfer from GBS-trained groups). Sex and phasing REC (Saaz) n/a REC (Saaz in the 09-24 / 09-25 records)
DArT genotypes (Pilgrim) REC: sync/2026-10-07/pilgrim_in_atlases/ (p01–p05, PIPELINE.sh, APPLY.sh) Lineage NP (no reads). Sex REC (DArT sex, p05). Phasing REC (progeny-inferred lanes, p03_parents.py); xconc skips it n/a NP: companion page only (p06, p07)
RNA-seq (Wye RNA lane) REC: o13_rna_pileup.sh → o03 / o04 → o15_wyerna_export.py, o16_rna_eval.py Lineage ND (not in COH). Sex lane and phasing not found n/a ND
VCF only (no BAM) REC: route V (clade_ab_imputation 07_extract_cohort.py → 08_impute.py → m04; divpanel, USDA, Havill, Saaz) Lineage NP as built: the layers count reads at a site list from BAMs n/a REC: USDA depth from FORMAT/AD (sync/2026-09-24/per_chromosome_dosage/), not in the served atlas
Genotype calls only (Zhang 2017 HapMap) REC: opt-in dataset since 2026-10-08 (sync/2026-10-08/zhang2017_hapmap_project/ancestral_haplotypes/scripts/zhang2017_atlas_export.py, tools/zhang_regroup.py); counting-rule bin calls, 5 Mb + 2 Mb only; files multires/*.zhang2017.json, loaded only when selected in the Datasets drawer Lineage NP (no reads); "no subspecies calls for this cohort" line n/a ND
HiFi / assembly FASTA Research step: assemblies define the diagnostic catalogue, and changing it invalidates all calls (components/INTROGRESSION.md §4.4) same Inputs REC: /data/hop_data/cascade_dovetail_atlas/report/PIPELINE.sh. Adding the row is DS: manifest entry in annevo_eval/scripts/atlas_extra_assemblies.json + atlas_deploy.py build/test ribbon. Repeat, ancestry and F-marker tracks ND. Carey 2026 assemblies ND New reference REC (09-25 00a / 00b; mr_config.py)
Hi-C ND ND Indirect only: Hi-C scaffolds the assemblies (annevo_eval/docs/APOLLO_ASSEMBLY_PROCESS.md, hop-karyotype-hic skill) ND

New sample into an existing cohort. Not done, by design (PI, 2026-10-08): a cohort is one experiment delivered in aggregate, so new samples always form a new cohort with its own fit (ingest/README.md D1). The notes below describe the current state only. - Dosage: the sample goes into the 09-25 manifest and everything is re-run. - Introgression: the EM model is fitted once per cohort at 5 Mb (sync/2026-09-30/multires_clade/README.md), and nothing says whether adding samples should refit it. - Pedigree: a new cross needs one CROSS line in p01_pedigree_build.py; atlas_layers.py pedigree is DS. - Cross parents: hard-coded in several places (technical doc §1.3).

Per-reference calls (Reference selector, 2026-10-13). A cohort is shown on a reference other than 21110M only when its NA/EU calls were re-made there (never lifted). Path: BAMs of the cohort on that reference on PBSS → annevo_eval/methods/per_frame_ancestry/ (scripts/run_frame.sh <frame> <paf_dir> <cohort> <bam_prefix_template> <GBS|WGS>; README "Extending to other cohorts") → add the cohort to COHORTS in hop_atlas/tools/build_frames.py → build_frames.py + patch_frames.py on a stage → deploy (components/INTROGRESSION.md §8e). Today: Zenith x 21058M on 5 references; other cohorts in progress.

Cohort manifests (2026-10-08). hop_atlas/cohorts/<cohort>/{cohort.yaml,samples.tsv} exist for apollo, carey2026, cascade, cordifolius, divpanel, havill, havill_blocks, saaz and wye (back-filled, except Carey, written fresh). ingest/pipeline/ingest.py validate|jobs|show reads them; the jobs generator reproduces all 30 historical job tables. OSU, USDA, Pilgrim, Wye RNA and Zhang 2017 have no manifest yet.

Steps common to every data type

Step Generic? Where
Download (SRA / ENA / PBSS) No, records only OSU /data/hop_data/osu_PRJNA1424574/scripts/DOWNLOAD_COMMANDS.sh ("not meant to be re-executed blindly"); Carey sync/2026-10-08/carey2026_hop_wgs/PIPELINE.sh + scripts/lepton/sra_stagein_worker.sh; CLAUDE.md data rules (PBSS first on EOS)
Read QC (fastp) Partly: inside the alignment job cluster/lepton_linear/linear_job.sh (DO_FASTP); annevo_eval/docs/ASSAYS.md
GBS enzyme and barcodes Rules only, no tool ASSAYS.md §1.5 (read the enzyme off the data), §1.6 (inline barcodes); one-off sync/2026-10-02/cascade_relatedness_final/scripts/03_divpanel_barcode_qc.py
Alignment DS cluster/lepton_linear/README.md, config/<pop>_<ref>.env, submit_linear.py; ASSAYS.md §4.2b; lepton / eos-slurm / parabricks skills
Duplicate rule DS (rule + gate) cluster/lepton_linear/README.md: GBS --no-markdups, WGS markdup, gate samtools view -c -f 1024
Stage / test / deploy DS tools/atlas_deploy.py, tools/atlas_layers.py

Gaps, in the order they block new data

  1. No generator for the base introgression atlas. Allele counting → NA/EU calls → viewer files are copy-and-edit per cohort. This blocks every new read-based cohort. Carey 2026 got in (2026-10-09) through cohort-specific stage scripts (cohorts/carey2026/pipeline/), the closest thing to a template so far.
  2. Cohort manifests exist but are not yet read by the builders. 9 manifests + loader + jobs (2026-10-08); the exporters still use hard-coded lists: COH / LIMITED_COH, COHORTS in build_subspecies_evidence.py and build_frames.py, about 5 dosage builder files, and the ribbon haplotype lists.
  3. No generic download or QC tool. Per-cohort records only.
  4. Dosage has no add-cohort procedure. The generic hop_adosage package is only on the unmerged branch dosage-methods.
  5. Ribbon:
  6. new-assembly inputs come from Cascade-specific scripts only;
  7. new rows get no repeat, ancestry or F-marker tracks;
  8. the legacy alignment can't be re-run (/workspace/joint_paintings_work missing);
  9. the build scripts are not in git.
  10. RNA-seq: not in the lineage layers or the dosage atlas. Hi-C: no role in any atlas.
  11. Stale doc: technical doc §10.2 says the OSU scripts folder is empty; it now holds o00–o18.

The fix: one effort closes gaps 1–3 and most of 4. Build a cohort manifest (accessions, data type, enzyme, parents, roles, truth plants) and one parameterised script per stage that reads it: download → QC → align → count → call → layers → viewer files. Then adding a cohort means writing a manifest and running a pipeline, not forking scripts. Effort: days to a few weeks, starting with the introgression base atlas because Carey 2026 is waiting on it.