Hop atlas web app: changelog

Dated record of changes to what is served. All components are tentative. The newest entries are first. Entries before 2026-10-08 are reconstructed from sync records and backups; their sources are given.

2026-10-13

Docs + packaging (PI request, ~17:45): documentation brought up to date (README components / layout / rules / open items; components/INTROGRESSION.md overview, patch-marker table and "How haplotypes are assigned"; ARCHITECTURE.md progress table; ADDING_DATA.md and ingest/README.md for Carey 2026 live, Zhang opt-in, manifests, per-reference calls); docs re-rendered (new /hop_atlas/docs/BUNDLE.html). New hop_atlas/tools/package_atlas_bundle.py (+ bundle_assets/, guide components/BUNDLE.md) builds the self-contained introgression-atlas zip with a generated CLAUDE.md. No atlas page or data changed.
Introgression (PI request, deployed 16:58; backups *.bak_2026-10-13_1658_frames): the "Display coordinates" control is now Reference (?ref=<key>, default 21110M), and the old liftover is removed.
Removed: the "Universal (consensus uMb)" option, the 14 lifted display frames, their drawing code, the liftover coverage table, and the lifted geometry in the served data (atlas_native_<res>.json per-bin g; atlas_data.json len/utile/notes.frame of non-21110M frames, ulen, coverage).
Listed references (only those with calls): 6 of 17 in the registry — 21110M male composite (served calls, all cohorts), Saaz hap1 (principal), 10-12 hap1 (principal, + X/Y), Apollo hap1, Apollo hap2, Cascade (HopBase Dovetail; listed because it has calls; orchestrator decision, easy to drop in build_frames.py). Each non-21110M reference carries the Zenith x 21058M calls re-made on it (126 plants; record sync/2026-10-13/per_frame_ancestry/).
On a non-21110M reference only the plants with calls on it are drawn, in its own bins and lengths; search says "no calls on " for others. 21110M-only layers (haplotype rows, AIM, phase lanes, badges, transmission, evidence tracks, sex-aware X/Y, haplotype-resolution levels, mixture/concordance filters, similarity sorts, site-level detail) are greyed with "available on 21110M only". ChrX is drawn and labelled "not sex-aware".
Registry: frames/frames.json (17 entries: 21110M composite, 13 panel haplotypes with Saaz/HL1012 pseudo-haploid marked deprecated and never offered, Saaz hap1, 10-12 hap1, Cascade). The per-frame Saaz hap1 / 10-12 hap1 are their own entries (not the panel sequences).
Tools: hop_atlas/tools/build_frames.py, patch_frames.py (/*frames*/, 40 hooks), frames_assets/frames.js; guide components/INTROGRESSION.md §8e.
Checks: page and data HTTP 200; headless: no script errors on 21110M, Saaz hap1, 10-12 hap1, Cascade, Apollo hap2, ?sample=; 21110M canvas byte-identical to the served 21110M frame before the change (5 Mb, 500 kb); default canvas probe paints. Suites not run.

2026-10-12

Introgression: the Havill cross is now labelled "Zenith x 21058M" everywhere it is displayed (PI).
Section headers: "Zenith x 21058M · parents/progeny".
Parent rows: "Zenith (mother)" and "21058M (father)".
Also: the pedigree section, filter messages, and the badge page and hover text.
Unchanged: the internal cohort key havill and sample keys, so links and data keep working.
Tool: hop_atlas/tools/relabel_havill.py (idempotent; 1,045 replacements); the evidence-page builder sources were updated too.
Backup: *.bak_2026-10-12_1750_zenith_label.

2026-10-10

Introgression: the "WGS subspecies (variants)" level now hides its unvalidated lupuloides/neomexicanus/pubescens guess (40–45 % correct on synthetic controls) by default (PI). It has its own fill flag, default off; the other levels keep fill on, and the checkbox follows the selected level. Patch hop_atlas/tools/patch_wgsv_fill_default.py (/*wfill*/). Backup lineage.js.bak_2026-10-10_1858_wgsv_fill.
Phylogeny (new component, PI request, deployed ~18:05): /phylogeny/, served from scratch_plots/phylogeny → /data/hop_data/hop_atlas/phylogeny/ (2.8 MB).
What it shows: the 753-tip pseudo-phylogeny (692 ApeKI-GBS backbone plants + 61 placed tips; Zhang 2017 not drawn), circular or linear, midpoint display root labelled as not inferred; highlighted founders / wild groups / 13 assemblies; colour rings (subspecies label, assay, cohort, atlas category); bootstrap threshold; collapsed clades named by contents; fuzzy search with zoom; hover/click details incl. placement stability; links to the introgression-atlas row and badges. Second panel: WGS joint-call BIONJ/ML trees (30 plants), catalogue WGS-only tree, per-chromosome/per-window trees; local-affinity heatmap with a window slider.
Build: hop_atlas/tools/build_phylogeny_view.py (page source hop_atlas/tools/phylogeny_view/); guide components/PHYLOGENY.md. Source record sync/2026-10-10/hop_phylogeny/.
Nav / landing: "Phylogeny" added to www/nav.js and a card to www/index.html (backups *.bak_2026-10-10_phylogeny).
Introgression: ?sample=<key or name> opens that sample's row (hook /*urlsample*/, tools/patch_url_sample.py; stage/deploy, backup index.html.bak_2026-10-10_1804_2026-10-10_url_sample). Used by the Phylogeny tip links.
Checks: HTTP 200 for the pages and data; Phylogeny page renders in headless Chromium with no script errors; suites not run.
Introgression (PI request, deployed 17:49 + legend-text fix 17:50; backups *.bak_2026-10-10_1749_wgs_variants, *.bak_2026-10-10_1750_wgs_variants_legendfix): new haplotype-resolution option "WGS subspecies (variants: lupulus/lupuloides/neomexicanus/pubescens/cordifolius)" (level key Lw).
What it shows: the 29 WGS rows of the 30-sample genome-wide joint call (Wye Target, BL2/1, Cascade cross parents, Apollo cross parents, 22 Carey 2026 plants, KR); every other row is drawn pale grey and its hover says "WGS-only level".
Model: five reference groups built from WGS plants only (incl. pubescens and cordifolius), scored at joint-call SNPs with REF and ALT read counts, kin-level leave-one-out; per native 21110M bin at 5 Mb / 2 Mb / 1 Mb / 500 kb.
Call = coarse pair state lupulus / American / cordifolius / U (grey = none of the five): synthetic-F1 controls could not separate lupuloides, neomexicanus and pubescens per bin with these reference plants. With "fill" ticked, a called American copy shows the most likely subspecies faded (unvalidated guess; pubescens = yellow-green).
Hover: row tile = call, posterior, informative SNPs, gate, guess; Evidence-detail = coarse and fine posteriors, observed vs expected read classes, per-group near-fixed SNP match, groups evaluable, gate, groups-v1b line.
Data: wgsv/ → /data2/hop_data/wgs_subspecies_variants/viewer_wgsv_20261010 (merged into the lineage JSON at load; the lineage/ data dir is unchanged). Hooks /*wgsv:1..16*/ (index.html + lineage.js) + wgsv.js (hop_atlas/tools/patch_wgsv_level.py).
Checks: page HTTP 200, no script errors, level merged for 29 rows, WGS and non-WGS hovers render; suites not run. Record sync/2026-10-10/wgs_subspecies_variants/.

2026-10-09

Introgression (PI report, display-only fix, deployed 22:19; backup lineage.js.bak_2026-10-09_2219_halves_phase): tile halves in the haplotype-resolution views (NA/EU BAM re-call, Subspecies, Lineage, Subspecies + cordifolius) are now ordered for display.
Cause: each bin's call is an unordered pair stored alphabetically, and the tile drew group 1 on top. A run of lupuloides/lupulus (orange on top) followed by lupulus/neomexicanus (blue on top) swapped halves, so parent rows looked lane-hopping.
Rule 1 (cross parents): where the parent's h1/h2 lanes are A/B with a phased confidence (assembly or progeny-linked a/l/m) and the pair is one clade-A plus one clade-B group, h1's group goes on top and h2's below. Otherwise rule 2.
Rule 2 (all rows, and the fallback): each group keeps the half it held in the previous called heterozygous bin of the same chromosome (greedy left to right; same group > coarse/fine version > same clade; ties keep the stored order).
Unchanged: every call, string and data file. The hover still gives the unordered pair, marked "(unordered pair; tile halves ordered for display)"; the legend says so.
Hooks: /*hphase:1..6*/ in lineage.js (hop_atlas/tools/patch_halves_phase.py). Before/after screenshots: sync/2026-10-09/halves_phase/figs/. Checks: page loads with no script errors; suites not run.
Introgression (PI request, deployed 21:42; backups *.bak_2026-10-09_2142_cordifolius_group): new EXPERIMENTAL haplotype-resolution option "Subspecies + cordifolius (experimental)".
What it shows: cordifolius is a 4th subspecies-level reference group: KR plus Carey CORDI 2/4/7/8, 5 different plants (identity check in the record). It is not tied to the NA/EU clade.
Display: magenta where a cordifolius copy is called (4-group tag-space re-call tagspace/calls/<coh>_t4c); the Subspecies tile everywhere else.
Unchanged: the existing NA/EU, Subspecies and Lineage options and their data are identical.
Data: lineage/ → viewer_lineage_20261009_cordifolius (adds L1c), subevc/ → viewer_subspecies_evidence_20261009_cordifolius.
Hooks: /*cordi:1..9*/ in lineage.js and /*cordi:s1..s8*/ in subev.js (hop_atlas/tools/patch_cordifolius_level.py). The builder has a new --modes tagc.
Headline: 3,646 of 511,158 called 5 Mb bins (774 of 2,116 non-cordifolius plants) carry a called cordifolius copy (1 Mb: 1,911 bins). Controls: leave-one-out 100 %; 20 shuffled fake groups give 33–61 k bins.
Checks: HTTP 200 only; suites not run. Record sync/2026-10-09/cordifolius_lineage/.
Introgression: subspecies/lineage evidence in the info boxes now covers Carey 2026 (22 plants × 4 resolutions).
Export: build_subspecies_evidence.py --cohorts carey2026; carey2026 added to the builder with the apollo_bam parameter set, as KR and the Carey calls use.
Placement: files copied into the served evidence dir viewer_subspecies_evidence_20261009_full without overwriting; run settings merged into its build.json (backup build.json.bak_2026-10-09_pre_carey).
Check: every L1 bin matches the stored call. 21 of 375,399 bins differ, all uncalled L2 ties (posterior ≤ 0.09).
Introgression (deployed 2026-10-09 18:57, PI-approved; backups *.bak_2026-10-09_1857_carey2026_20261009c): Carey 2026 WGS panel (PRJNA1437418, 22 plants) built on stage /data/hop_data/lineage_refinement/stages/carey2026_20261009c from the current served atlas (includes /*sclear*/ and /*subev*/).
Sections: five, one per paper variety label: "Carey 2026 · Cordifolius / Neomexicanus / Lupuloides / Pubescens / Lupulus (paper label)", with 4 / 5 / 3 / 3 / 7 plants.
Caveat: every Carey section header says the two-state NA/EU fit is not established for the non-lupulus plants (as for cordifolius KR).
New preset "Wild WGS (Carey 2026 + KR)" (?preset=wild_wgs), 90 rows: the 15 non-lupulus Carey plants, KR, and divpanel "H. neomexicanus" / "wild-collected N. American" / "US wild" (21 / 20 / 33).
Datasets drawer: carey2026 is default ON.
Sex lane: MSY reads (Apollo WGS mask and valley) and panel-mode pan-X k-mer dosage (untested mode) must agree. Result: 7 males (6 Am-Y, 1 L-Y), 11 females, 4 not verified (diploid X, flagged). The paper's sex labels are labels only; no plant is called against its label.
Lineage layer: Carey ChrX now sex-aware.
Checks: stage == served except Carey (0 failures); x-strings == call tables (0 mismatches at 4 resolutions); canvas probe: default 2,299 sample rows at 29,551 px, painted; wild preset 90 rows, painted.
Deploy: atlas_layers.py deploy <stage> --yes. Record: /data/hop_data/cohorts/carey2026/HANDOFF_INTROGRESSION.md §8, sync/2026-10-09/carey2026_hop_wgs/.
Introgression (PI-requested, deployed 18:21): subspecies / lineage evidence in the Evidence-detail hover boxes. In the Subspecies and Lineage views, a calling-bin hover in the detail panel now continues below the NA/EU box with the subspecies evidence, laid out the same way. It shows:
the call line: state, posterior, informative tag loci, the thresholds (posterior ≥ 0.95, ≥ 3 loci), or "forced" by the parent call;
the candidates allowed by the NA/EU call;
observed loci with REF only / ALT only / both, against the expected counts and posterior for each candidate state;
per group (lupulus / lupuloides / neomexicanus): the reference ALT frequency, reference plants per locus, and loci matching the group;
the light-red / faded explanation;
on ChrX, the sex-aware (haploid) note.

The Lineage view adds the European-lineage block (SNP layer v6). Zhang 2017 and Pilgrim rows say "no subspecies calls for this cohort (no reads)". The numbers are re-derived from the callers' own inputs and settings, with no new calls. Of 35.5 M exported bins, all but 2,468 match the stored call exactly; those 2,468 are uncalled L2 bins (posterior ≤ 0.469) where several states tie on posterior and the tie was broken differently. Data subev/ → /data/hop_data/lineage_refinement/viewer_subspecies_evidence_20261009_full (2,098 samples × 4 resolutions, 2.8 GB). Tools build_subspecies_evidence.py, patch_subspecies_evidence.py (/*subev:1..4*/), subev_assets/subev.js. Feature check on the stage 33/33 (incl. must-fail: no subspecies fields at the NA/EU level; Zhang row line). Canvas probe: default view unchanged (2,372 rows, painted). Viewer suites not run. Backup *.bak_2026-10-09_1821_subspecies_evidence3. Record sync/2026-10-09/subspecies_evidence/. - Introgression: the sample search can be unset (PI). A Clear button next to "Show sample", and Esc in the search box, empty the search, forget the selected sample and return to the previous view. Patch hop_atlas/tools/patch_sample_clear.py (marker /*sclear*/, idempotent). Checked on the stage: Clear → panels (2,277 rows); Esc → previous view; no page errors; default view still paints. Backup *.bak_2026-10-09_*_sample_clear.

2026-10-08

Introgression + nav (PI-requested, deployed 23:47): badge evidence pages and graded paternity badge.
Badge index: evidence/index.html, with one entry per badge type (prose by Fable, hop_atlas/evidence/BADGES.md).
OSU paternity evidence page: evidence/osu_paternity.html?s=<seedling> (prose hop_atlas/evidence/osu_paternity.md), with ranked OH and flip statistics with block-bootstrap intervals, histograms, the evidence strip along the genome, atlas bins vs SNP evidence, and a card for the clicked seedling.
Graded badge: father≠19058m · <grade> on 73 OSU seedlings (strong 67, moderate 5, possible 1; data from sync/2026-10-08/osu_paternity_uncertainty/).
Hover pop-up on every row badge, with a See evidence button.
Nav bar: "Badges" entry added; "Introgression" no longer highlights on the evidence pages.
Re-applied on top of the Zhang deploy: with build_paternity_badges.py, patch_evbadge.py and build_evidence_pages.py.
Checks: load check and canvas probe (default view unchanged; ?preset=not_19058m 72 rows, painted). The feature test passed 28/28 on the first stage. Backups *.bak_2026-10-08_2347_evidence_badges2.
Introgression (PI-approved, deployed 23:46): Zhang 2017 Hopsteiner HapMap added as an opt-in dataset (zhang2017, 4,453 GBS libraries, 37 sections, 5 Mb + 2 Mb). New Datasets drawer: only selected cohorts are drawn; default = all except Zhang; selection in ?ds= and per browser. Presets preselect their cohorts; new preset "Zhang 2017 HapMap". Zhang rows load only when selected (+18.5 MB at 5 Mb). New canvas guard in every view folds the largest sections (or clamps) instead of drawing a blank canvas past the 32,000 px browser limit. Default view unchanged (2,372 rows). Tools: zhang_regroup.py, patch_datasets_guard.py, probe_canvas.py; record sync/2026-10-08/zhang2017_atlas_ingest/. Probe on :8900: default view unchanged (2,372 rows, 29,187 px, painted, 9.3 MB); ?preset=zhang2017 2,415 Zhang rows, 30,591 px, painted, 27.8 MB. Backups *.bak_2026-10-08_2346_zhang2017_hapmap. Known: ?ds=all is not recognised (empty view). Record sync/2026-10-08/zhang2017_atlas_ingest/.
Introgression: Saaz non-progeny exclusion unified (12 samples; PI-approved). DRR628685–690 (deposited cultivars) and DRR613305/306/307/388/435/436 are now excluded from every progeny-based computation, from one manifest (hop_atlas/cohorts/saaz/cohort.yaml, scope progeny) read through hop_atlas/ingest/pipeline/exclusions.py; 14 files in 13 recurrent scripts corrected. Served changes are Saaz only (20 files): 6 cultivars relabelled non-progeny; transmission / B-frequency on 243 plants (was 249); 500 kb phase blocks + concordance (55 plants changed; 1–5 Mb unchanged); Saaz similarity-sort panels re-clustered; pedigree progeny node 249 → 243. Every non-Saaz entry byte-identical (349/349). Exclusion test 8/8 incl. must-fail; load check 21/21 HTTP 200, no 404s; viewer suites not run (PI: on request). Backup *.bak_2026-10-08_2215_saaz_excl2. Record hop_atlas/ingest/SAAZ_EXCLUSIONS.md, sync/2026-10-08/saaz_exclusions/.
Introgression: sex-call disagreement badges for 4 Saaz plants (DRR613202, 213, 254, 332): the 09-24 MSY depth caller calls them male (unresolved), the 10-05 k-mer scan female (two X, no Y-specific k-mers). Badge only; X/Y lanes unchanged (PI). Cascade's equivalents have been badged since 10-04. Filter sex:disagree = 138 rows. v05 30/30 on stage and :8900; 7/7 suites on stage. Backup *.bak_2026-10-08_2129_saaz_sexbadge. Record sync/2026-10-08/saaz_sex_badges/.
Docs: ingestion runbooks per data type in hop_atlas/ingest/ (GBS ×3 kinds, WGS, GRAS-Di, RNA-seq, assemblies), reconstructed from the dated records, plus ingest/README.md (common stage skeleton, rules, PI decisions D1–D7, record errors, pipeline plan). Rendered at /hop_atlas/docs/INGEST.html. components/DOSAGE.md corrected: hop_adosage was merged in 029bf81.
Navigation: the landing page is now the :8900 root page (/). The former bare directory listing of scratch_plots/ is replaced by /hop_atlas/files.html. A shared nav bar was added to every page of all three atlases and the docs via stage → test → deploy: introgression 7/7 suites, ribbon 7/7, dosage 3/3; check_nav.py 11/11 pages on :8900. Backups *.bak_2026-10-08_1908_nav_2026-10-08.
Docs: ADDING_DATA.md, a data type × atlas audit of the workflows for adding samples (answer: no generic workflow yet; gaps listed).
Ribbon: similarity view structural_atlas/similarity.html (ranking / one vs all chromosomes / matrix, from the existing pair_distances_v2.tsv; no recomputation). Tests: 12/12 plus the 6 existing verifiers on the stage, then deployed (backups *.bak_2026-10-08_1834_similarity_2026-10-08).
Cleanup (PI-approved):
review servers :8902 and :8905 stopped;
unused stage folders removed (/data/hop_data/lineage_refinement/stage_viewer{,_v2,_v3,_v4}, lineage_refinement/stages, /data/hop_data/atlas_stages; about 4 GB), each logged as removed in data_products.tsv. They were checked first: no served symlink, process or reference used them, and the served viewer_lineage is byte-identical to the v2 copy;
test defaults now point at :8900.
Web app folder created: hop_atlas/ (this folder) for the three components.
The guides moved here from annevo_eval/docs/*_ATLAS_UPDATES.md (stubs left).
The tools moved here from annevo_eval/scripts/ (forwarding stubs left).
Landing page at :8900/hop_atlas/.
Dosage atlas moved out of sync/ onto :8900 (/dosage_atlas/).
Self-contained home scratch_plots/dosage_atlas/.
The four former sync/ folders were moved byte-for-byte to /data/hop_data/hop_atlas/dosage_sources/ (checksum manifests; .MOVED.txt pointers).
The builder was repointed to the archive.
:8906 stopped.
The broken family_shared_site_dosage link was fixed.
The Pilgrim box is now preserved across rebuilds.
Tests: bundled 3/3 + HTTP smoke 4/4 on :8900.
Introgression atlas:
pedigree lookup (?ped=);
"how these layers are updated" links;
before that (same day) the haplotype-resolution layers went live, rebased onto the Pilgrim version. Backups *.bak_2026-10-08_prelineage, *.bak_2026-10-08_prepedigree.
Tests l23–l27, p11 and p12 pass on :8900.
Tooling: atlas_deploy.py (all components) and update guides for the ribbon and dosage atlases. Ribbon 6/6 verifiers and dosage browser checks pass on stages, with must-fail controls.

2026-10-07

Introgression: Pilgrim × 316/01/10 DArT cohort and the pilgrim_wye preset (source: sync/2026-10-07/pilgrim_in_atlases/).
Dosage: Pilgrim companion link box added to the v2 page by hand (backup START_HERE.html.bak_2026-10-07).

2026-10-05

Dosage: OSU sex-chromosome panel (source: sync/2026-10-05/dosage_explorer_v2_sexpanel/).

2026-10-04

Dosage: explorer v2, ancestry-aware copy number (source: sync/2026-10-04/dosage_explorer_v2/).

2026-10-03

Introgression: OSU v2 phasing, the Y/X lane, the Wye RNA lane and the evidence-track rebuild (source: catalog entry osu_PRJNA1424574_introgression_atlas; backups *.bak_2026-10-03*).
Ribbon: Cascade HopBase row, candidate loci, genome-wide views and contig junctions (sources: structural_atlas/PIPELINE_{CASCADE_ROW,CANDIDATES,GENOMEWIDE,JUNCTIONS}.sh).
Server: :8900 switched to the no-cache server.

2026-10-02

Introgression: OSU PRJNA1424574 cohort (source: sync/2026-10-02/osu_introgression_atlas/; backups *.bak_2026-10-02_osu).

2026-10-01

Ribbon: atlas v2 (source: structural_atlas/PROVENANCE_V2.md).

2026-09-28 / 09-29

Dosage: explorer v1 (source: sync/2026-09-28/dosage_explorer/, now archived).
Introgression: universal / multi-resolution introgression atlas (source: annevo_eval/docs/INTROGRESSION_ATLAS_TECHNICAL.md).