Hop atlas home · Overview · Changelog · Architecture · Adding_Data · Introgression · Ribbon · Dosage · Phylogeny · Bundle · Ingest · Ingest_Gbs · Ingest_Wgs · Ingest_Grasdi · Ingest_Rnaseq · Ingest_Assembly

Rendered from hop_atlas/components/INTROGRESSION.md (2026-10-13)

Introgression atlas: how to update it (pedigree, subspecies / lineage layers, new cohorts)

Status: living document, started 2026-10-08. All products it describes are tentative. Served atlas: scratch_plots/introgression_atlas, on port 8900 (http://10.112.218.205:8900/introgression_atlas/). Other atlases: the ribbon (structural) atlas and the dosage atlas have their own guides, RIBBON.md and DOSAGE.md. All three share the generic stage / build / test / deploy tool hop_atlas/tools/atlas_deploy.py (profiles introgression, ribbon, dosage). For this atlas, atlas_layers.py (below) adds the layer-specific build steps. Companion: annevo_eval/docs/INTROGRESSION_ATLAS_TECHNICAL.md describes how the base atlas is computed (catalogue, NA/EU calls, phasing, sex chromosomes, front end). Its snapshot is dated 2026-10-03. This document covers what was added later and how to update it.

Overview: what is served today (2026-10-13)

The atlas is one page (index.html) drawing every row on a canvas; one row = one sample library, one tile = one calling bin (5 Mb / 2 Mb / 1 Mb / 500 kb). Everything is static JSON; the page grew by anchored patches (table below).

Rows. 2,299 sample rows drawn by default, from 12 cohorts: diversity panel 199, USDA 173, Zenith x 21058M 130, Cascade x LB_male 182, Saaz x HL1012 257, Wye Target x BL2/1 144, Apollo x Pub_740M 186, KR (cordifolius) 1, OSU 826, Wye RNA lane 13, Pilgrim (DArT) 166, Carey 2026 WGS 22 (live since 2026-10-09). Opt-in: Zhang 2017 HapMap, 4,453 rows (5 Mb and 2 Mb only). Counts from multires/atlas_samples_5Mb*.json.

Haplotype-resolution levels (drop-down; each level is conditioned on the one above it):

Pilgrim (DArT) and Zhang 2017 (genotype calls) have no reads, so they stay on the base NA/EU lane in every level.

Controls and URL state. - Reference (?ref=<key>, default 21110M): only references with real calls are listed (today 21110M + Saaz hap1, 10-12 hap1, Apollo hap1, Apollo hap2, Cascade, for Zenith x 21058M). On another reference only the plants with calls there are drawn, in its own bins, and 21110M-only layers are greyed. The lifted ("liftover") display frames and the universal uMb option were removed on 2026-10-13 (§8e). - Datasets drawer (?ds=a,b,..., remembered per browser): which cohorts are drawn; Zhang 2017 is off by default and its files load only when selected. Canvas guard: past the ~32,000 px browser canvas limit the largest open sections are folded (or the canvas clamped) with a message, never a blank canvas. - Presets (?preset=<id>): saved queries that switch on the datasets they need, e.g. zhang2017, wild_wgs ("Wild WGS (Carey 2026 + KR)", 90 rows), not_19058m, pilgrim_wye. - ?sample=<key or name> opens a row (used by Phylogeny tip links); ?ped= opens the pedigree lookup. The sample search has Clear and Esc. - Badges and evidence pages. Row-label badges (sex disagreement, trace Y, OSU tags, graded paternity father≠19058m · strong|moderate|possible) open a popover with See evidence, linking to the Badges page evidence/index.html or the OSU paternity page (§7). "Badges" is in the nav bar. - Evidence detail (click a row): per-bin read evidence; in the Subspecies / Lineage levels the hover box continues with the subspecies evidence (§8). - Labels: the Havill cross is displayed as "Zenith x 21058M" (cohort key havill unchanged; tools/relabel_havill.py, 2026-10-12).

Patch markers

Each feature added after the base page is an anchored, idempotent patch: the script asserts every anchor occurs exactly once, skips hooks already present, and writes a marker comment so the hook can be found. Found in the served index.html and *.js on 2026-10-13 (grep -o '/\*[a-z0-9-]*' index.html *.js); tools/ = hop_atlas/tools/. The packager (tools/package_atlas_bundle.py) re-scans the markers each time it runs and warns about any it cannot map to a script.

Order after a re-copy. If a cohort script rewrites index.html or atlas_layers.py patch re-copies lineage.js, re-apply on the stage in the order they were first deployed, which reproduces the anchors each script was written against: atlas_layers.py patch, patch_datasets_guard.py, patch_evbadge.py, patch_sample_clear.py, patch_subspecies_evidence.py, c26_23_patch_index.py, patch_cordifolius_level.py, patch_halves_phase.py, patch_wgsv_level.py, patch_url_sample.py, patch_wgsv_fill_default.py, relabel_havill.py, patch_frames.py. Each --check lists its hooks; a script whose anchor is missing aborts without writing.

How haplotypes are assigned (plain words)

Common recipe. For every level: (1) pick sites where reference groups of plants or assemblies carry different bases; (2) count each sample's reads carrying each base at those sites; (3) for each bin, compute how likely the reads are under each pair state (AA / AB / BB, or a pair of subspecies); (4) call the bin only if the best state has posterior ≥ 0.95 with ≥ 3 informative sites, otherwise leave it white. Bins are called independently (no smoothing between bins). A tile is an unordered pair; its two halves are ordered for display only (§8c).

Autosomes, NA/EU. - Clades are defined by assemblies only: in each 5 Mb bin, each of the 13 panel haplotypes is called clade A (European-like, close to drHumLupu, Saaz, HL1012) or clade B (American-like, close to the two wild American plants MN1421 and MN586). - Diagnostic sites: positions where ≥ 3 clade-A haplotypes all share one base and ≥ 3 clade-B haplotypes all share a different base, about 29 M sites on 21110M, thinned to one per 150 bp grid cell. - Counts: reads aligned to 21110M, piled up at those sites (route B: BAM pileup; MAPQ ≥ 20, base quality ≥ 20; GBS duplicates kept, WGS duplicates dropped). - Model: per cohort, a two-state EM model fitted once at 5 Mb and held fixed at finer resolutions. Besides a read-error term it has an outlier term (paralogous reads), separate dropout terms for a het that looks AA or BB (GBS restriction-site and library effects), and a B-read efficiency w that absorbs reads carrying the non-reference base mapping less well.

Subspecies and lineage. - Subspecies (L1): reference groups of plants (e.g. 28 lupuloides and 35 neomexicanus GBS plants, plus lupulus), each plant scored leave-one-out; counted in alignment-free ApeKI cut-site tag space (64-mers), so it does not depend on the 21110M alignment. Conditioned on the NA/EU call (an AA bin can only be lupulus / lupulus). - Lineage (L2): European lineages on the lupulus copies from the SNP layer, conditioned on L1; American copies stay at subspecies level. WGS and GRAS-Di cohorts get subspecies only, because the GBS-trained lineage calls do not transfer. - Cordifolius (L1c, experimental): a 4th group (5 plants) allowed under any NA/EU call. - WGS variants (Lw): five groups built from WGS plants only (incl. pubescens and cordifolius), scored at joint-call SNPs. Synthetic F1 controls could not separate lupuloides, neomexicanus and pubescens per bin, so only lupulus / American / cordifolius / none is called; the finer guess is unvalidated and hidden by default.

Sex chromosomes. - ChrX: sex-aware. Males carry one X copy in the X-specific region (below ~234 Mb on 21110M ChrX), so only homozygous (haploid) states are possible there; 234–240.8 Mb depends on the Y lineage; the pseudo-autosomal region is diploid, with reads summed over X and Y. - ChrY: the male-specific region is called European-type (L) or American-type (Am) from sites where 21110M and the MN1421/MN586 Y differ. - Sex itself comes from male-specific-Y read depth (mother-masked, within a cross), cross-checked by X dosage and Y k-mers; disagreements are badged, not overridden. - On a non-21110M reference, ChrX uses one diploid model for every plant and is labelled "not sex-aware".

Reference-bias mitigations. - Read counts come from BAM pileups (reference-only reads counted), not variants-only VCFs, for every cross moved to route B; VCF cohorts hid reference-allele evidence and gave "depleted" all-AB bins. - The EM's dropout terms and B-read efficiency absorb systematic under-counting of the non-reference allele. - Subspecies calls use alignment-free tag space. - Leave-one-out scoring of reference plants; must-fail controls (shuffled groups, swapped labels) for each new level. - Per-reference calls (§8e; annevo_eval/methods/per_frame_ancestry/): the whole call (catalogue, counts, EM) re-made on other assemblies, so a call can be compared across references instead of trusting 21110M alone.

Biases that remain. - All served calls are made on one reference (21110M, a male European-type composite); per-reference calls exist so far for one cohort only. - Ascertainment: the diagnostic sites come from 13 assembly haplotypes; clade B rests on two wild American plants, and none of the assemblies is pubescens or cordifolius, so the two-state NA/EU fit is not established for those (Carey section headers and KR say so). - Small wild reference groups for the subspecies levels. - Zenith x 21058M, Saaz, divpanel and USDA base calls still come from variants-only VCFs (route V), which over-call AB (PI decision D2, ingest/README.md). - The catalogue is 5 Mb-derived; inside fine bins whose call switches, site density drops (~0.2×). - Genotype-only cohorts (Pilgrim, Zhang 2017) have no read depth: hets are under-called and no finer level is possible.

Short answer: can a new dataset go from SRA to the atlas by following written instructions?

No, not end to end yet. - Read alignment is generic and runnable: cluster/lepton_linear/ (README, linear_job.sh, per-population config/<pop>_<ref>.env). - The add-on layers have one tool: pedigree, subspecies / lineage, sex-aware ChrX labels and deployment use hop_atlas/tools/atlas_layers.py (§2–§5). - Everything else is cohort-specific: the steps between alignment and the served atlas JSON (allele counting, NA/EU calls, phasing, sex chromosomes, detail files, viewer patch) exist only as dated, copy-and-edit scripts. OSU used o01…o18, Pilgrim used p01…p05, Wye used m01…m07. - Download and read QC exist only as one-off records.

§6 maps every stage to the best existing script and marks each gap.

1. What is served, and where it comes from

scripts/ and viewer/ above mean sync/2026-10-07/lineage_refinement_v2/scripts/ and …/viewer/ in the main checkout.

2. The rule: stage → change → deploy (tests on request)

Tests run only when the PI asks (PI, 2026-10-08). They are kept and documented, but not run after every change.

Other sessions edit the served atlas too. On 2026-10-07 a Pilgrim cohort was added while the lineage layer was being staged. Never copy an old staged page over the served one. Always start from the current served copy:

T=hop_atlas/tools/atlas_layers.py
python3 $T stage --name my_change          # copies :8900 to /data/hop_data/lineage_refinement/stages/my_change/ + checksum snapshot
STG=/data/hop_data/lineage_refinement/stages/my_change
# ... make changes on the stage (sections 3 to 5) ...
python3 $T serve $STG --port 8907          # review at http://10.112.218.205:8907/introgression_atlas/
python3 $T test  $STG --port 8907          # ONLY when the PI asks: every viewer test -> $STG/checks.txt (~5 min)
python3 $T deploy $STG                     # refuses if :8900 changed since `stage`; lists files; asks; backs up *.bak_<date>_<name>
# python3 $T test /workspace/annevo-monkeyflowers/scratch_plots/introgression_atlas --port 8900   # only when the PI asks

• Deploy is safe by construction. It copies only changed or new files, the page last. Every file it replaces is backed up first, and it deletes nothing.
• Rollback: copy the *.bak_* files back. For the lineage data, point the lineage/ symlink back at the previous viewer_lineage* directory; the tool never modifies an old data directory.
• Changes to the served atlas need PI approval. Note the approval in the change's sync entry.

3. Updating the pedigree

Source: /data/hop_data/pedigree.txt, from doi:10.13020/02k6-6995. - Format: tab-separated, columns Selection Mother Father, with one header line. Lines starting with # are comments. - Conventions (kept as distinct plants, never merged with the base cultivar):

To add or correct a pedigree: edit pedigree.txt (keep a dated backup), then rebuild on a stage:

python3 $T pedigree $STG      # writes $STG/pedigree/atlas_pedigree.json and the review tables

Review the matches before deploying: - tables/P01_pedigree_matches.tsv lists every pedigree-name → sample match with its basis. - tables/P01_unmatched_nodes.tsv lists the entries without samples.

How names are matched (all in scripts/p01_pedigree_build.py):

• Adding a synonym: add it to SYN only when the two names are the same plant, not merely related. Petham Golding, for example, is deliberately not equated with any other Golding.
• A new mapping population needs one line in CROSS (cohort → mother, father, label).
• In the viewer, non-exact matches carry an asterisk in the tree, their basis in the hover, and "matched …" in the section label.

4. Updating the subspecies / lineage layers

4.1 What the layers are built from

4.2 Re-export after calls change for existing cohorts

python3 $T lineage $STG    # E03 strings + E04 per-bin evidence into a NEW /data/hop_data/lineage_refinement/viewer_lineage_<stamp>/
                           # and points $STG/lineage at it (the served data directory is untouched until deploy)
python3 $T test $STG --port 8907

l23_lineage_v3_test.py re-derives every exported string from the call files independently, so a bad export fails that test.

4.3 Add a cohort to the subspecies / lineage layers

These steps are recorded in sync/2026-10-07/lineage_refinement_v2/PIPELINE.sh. That file is documentation only (paths hard-coded), and each step is run per cohort. 1. BAMs. Mapped to 21110M, with no duplicate marking for GBS (see annevo_eval/docs/ASSAYS.md). 2. SNP layer. Allele counts at the L01 catalogue sites: sync/2026-10-06/lineage_refinement/scripts/l02_count_one.sh (l02_run_list.sh; WGS via l02_make_wgs_jobs.py / l02_run_wgs.sh on the 500 bp thinned target list), then l03_build_sparse.py. GBS keeps duplicates. Then c01_call_levels_v6.py → calls/<cohort>_v6. 3. Tag space. Extract the cut-site 64-mers with kt (samtools view -F 0x900 <bam> | kt --mode gbs5scan for GBS; --mode scan for WGS and GRAS-Di). Then t03_run.sh <cohort> (sparse counts) and EXTRA='--minref 1' c01t_run.sh _t4 <groups> <cohort> with --l0-from the v6 calls → tagspace/calls/<cohort>_t4. 4. Register the cohort in the export. Add it to the hard-coded COH list in e03_viewer_export.py, e04_detail_export.py and l23_lineage_v3_test.py. If the cohort's GBS-trained lineage calls do not transfer (WGS, GRAS-Di), add it to LIMITED_COH (§3 of the dossier explains why). 5. Export and test. atlas_layers.py lineage, then test.

Cohorts with no reads (Pilgrim, DArT genotypes) cannot enter these layers. They stay on the NA/EU atlas lane in every view.

4.4 Changing the groups or the catalogue

Changing the groups (C00_groups.tsv) or the haplotype catalogue invalidates all calls. Re-run steps 2–5 for every cohort, plus the separability tests (c02t_separability.py) and the population-validated A/B filter (n03_ab_site_filter.py, n06_holdout.sh). This is a research step, not routine maintenance.

4.5 Sex-aware ChrX labels

After phase/ or sexchrom/ change (a new cross, re-phased parents, new sex calls):

python3 $T xconc $STG     # X11: ChrX a/b/c + parental ChrX lanes; keeps the originals as *.atlas_orig.json; skips crosses without
                          # progeny in the concordance file or without sex-aware parent calls (e.g. Pilgrim)

5. Viewer code

• Code location: the layer code lives in viewer/lineage.js and viewer/pedigree.js. The hooks into the base page are exact-once string replacements: viewer/patch_index_lineage.py, then viewer/patch_index_pedigree.py. Each aborts if an anchor is missing or not unique, so a base-page change that breaks an anchor fails loudly.
• When to re-patch: if someone replaces index.html with an unpatched version (for example a cohort script that rewrites the page), run atlas_layers.py patch $STG. It re-applies whichever hooks are missing and copies both scripts.
• Tests (viewer/run_all_checks.sh <url> <atlas dir> <out>; each test includes a check that must fail if the feature is broken):

6. Ingesting a new cohort end to end: what exists today

Proposed next step (not yet done): a cohort manifest (accessions, assay, enzyme, parents, roles, truth plants) and one parameterised script per stage 1–8 that reads it. Stage 7 would then regenerate every served JSON from the manifests instead of patching. The technical doc §10.2 makes the same proposal.

7. Evidence pages and row-label badges ("See evidence")

What it is. Row-label badges (sex disagreement sex F, trace Y trY, OSU anomaly tags, and the graded paternity badge father≠19058m · strong|moderate, father≠19058m? · possible) open a popover on hover or keyboard focus. The popover shows the badge's evidence summary and a See evidence button: paternity → evidence/osu_paternity.html?s=<seedling>; every other badge → evidence/index.html?s=<key>#badge-<type> (the badge page). The atlas legend links to the badge page. Deployed to :8900 on 2026-10-08 23:47 (re-applied on top of the Zhang deploy); "Badges" is in the nav bar.

Prose tokens (each on its own line in the .md; the paragraph after a {{FIG:…}} token is its caption): {{SEEDLING_CARD}}, {{FIG:oh_ranked}}, {{FIG:flip_ranked}}, {{FIG:oh_hist}}, {{FIG:flip_hist}}, {{FIG:evidence_strip}}, {{FIG:atlas_vs_snp}}, and {{BADGE_TABLE}} (badge page; appended at the end when absent). A missing .md is built as a placeholder, so the build always runs. ?s=<seedling> highlights that seedling in every figure and fills the card; without it the page shows cohort-level views.

Build / rebuild on a stage (all idempotent; all take the atlas dir, so they re-apply to a re-staged page, e.g. after another session's cohort patch):

T=hop_atlas/tools; STG=/data/hop_data/lineage_refinement/stages/<name>/introgression_atlas
python3 $T/build_paternity_badges.py $STG     # vpg field (removes and rewrites all vpg; one-time backup *.bak_pre_evbadge)
python3 $T/patch_evbadge.py $STG             # hooks; aborts unless every anchor occurs exactly once; --check lists hooks
python3 $T/build_evidence_pages.py $STG      # pages + data; re-run whenever hop_atlas/evidence/*.md or the audit tables change
python3 $T/test_evidence_badges.py http://localhost:<port>/introgression_atlas/ $STG   # feature test (28 checks, incl. must-fail)

Adding a badge type: add it to BADGES in build_evidence_pages.py (catalogue row + anchor), to SELQ/typeOf in evidence_assets/evbadge.js, and describe it in hop_atlas/evidence/BADGES.md. A new evidence page = a new .md in hop_atlas/evidence/ (rendered as <name>.html); point the badge's popover link at it in evbadge.js.

Nav bar: "Badges" was added to www/nav.js on 2026-10-08 (www/nav.js is served live, so editing it changes :8900 immediately).

8. Subspecies / lineage evidence in the Evidence-detail hover boxes

What it is (PI request 2026-10-09). In the Evidence-detail panel (click a sample row), hovering a calling bin shows the NA/EU box (A only / B only / both, observed vs expected per state). When the haplotype-resolution selector is Subspecies (L1) or Lineage (L2), the same box now continues with the subspecies evidence for that bin, laid out the same way. Lineage view: the subspecies block, then the European-lineage block. At the NA/EU levels nothing changes. Deployed to :8900 on 2026-10-09 18:21 (stage /data/hop_data/lineage_refinement/stages/subspecies_evidence3; backups *.bak_2026-10-09_1821_subspecies_evidence3).

How the numbers are made. hop_atlas/tools/build_subspecies_evidence.py <atlas_dir> re-runs the callers' scoring (c01t_call_levels.py for L1, c01_call_levels_v6.py for L2) from the same sparse allele counts, catalogue, donor groups and settings (_run.json), and exports the intermediate quantities the calls were made from. It makes no new calls: every exported bin is compared with the stored call (state, pmax within 0.002, informative count), and the build writes checks_summary.tsv / checks_disagree.tsv next to the data. 2026-10-09 build: 35,504,311 bins, 2,468 disagree. All are uncalled L2 bins (posterior ≤ 0.469) with the same posterior and locus count, where states tie on posterior and the export (argsort kind stable) breaks the tie differently from the caller (default argsort). Two inputs changed after the calls were made; the builder uses the versions in effect on 2026-10-07: the plant-identity table (--identity, default sample_identity.tsv.bak_2026-10-08_saazsex) and the sex-chromosome JSON (the stage's served copy, which matches for these fields).

T=hop_atlas/tools; STG=/data/hop_data/lineage_refinement/stages/<name>/introgression_atlas
python3 $T/build_subspecies_evidence.py $STG --nproc 4    # ~15 min for 9 cohorts; NEW data dir + $STG/subev symlink
python3 $T/patch_subspecies_evidence.py $STG              # hooks + subev.js
# re-staged page, same lineage/ data: reuse a finished export instead of rebuilding
python3 $T/build_subspecies_evidence.py $STG --link-existing /data/hop_data/lineage_refinement/viewer_subspecies_evidence_<stamp>

When to rebuild: after the call files change (re-calling, or a new cohort registered in the export: add it to COH in build_subspecies_evidence.py as for E03/E04), or after atlas_layers.py lineage changes the bin layout. Feature check (with must-fail checks): sync/2026-10-09/subspecies_evidence/check_subspecies_evidence.py <url> <atlas_dir> <figdir>.

8b. Experimental level "Subspecies + cordifolius" (L1c, 2026-10-09)

What it is. A 4th L1 reference group, cordifolius, made of 5 plants: kr|KR and carey2026 CORDI 2/4/7/8. - Groups table: sync/2026-10-09/cordifolius_lineage/tables/C00c_groups_cordifolius.tsv. C00 is untouched. - Wildcard parent *: cordifolius is allowed under any NA/EU call, its donors count at every bin, and it is its own clade for shrinkage. - Re-call: all 10 read cohorts into tagspace/calls/<coh>_t4c, using the caller fork sync/2026-10-09/cordifolius_lineage/scripts/c01tc_call_levels.py. With the original C00 groups the fork reproduces _t4: 0 / 307,034 bins differ. - Viewer: shows the existing Subspecies tile, overlaid with every CALLED cordifolius-containing state.

Rebuild on a stage (after lineage or the calls change):

S=sync/2026-10-09/cordifolius_lineage/scripts
python3 $S/e05_cordi_viewer_export.py <current lineage data dir> <NEW dir>; ln -sfn <NEW dir> $STG/lineage
python3 hop_atlas/tools/build_subspecies_evidence.py $STG --modes tagc --link-name subevc --out <NEW evidence dir>
python3 hop_atlas/tools/patch_cordifolius_level.py $STG    # lineage.js /*cordi:1..9*/, subev.js /*cordi:s1..s8*/

Re-apply after a re-copy. Re-run the patch whenever atlas_layers.py patch or patch_subspecies_evidence.py re-copies lineage.js or subev.js: the hooks live in the atlas copies, not in their sources.

When a new cohort is added to E03/E04, also add it to COH in build_subspecies_evidence.py and re-run e05 and the tagc build.

8c. Display order of tile halves (/*hphase*/, 2026-10-09)

What it is. A haplotype-resolution call is an unordered pair state, stored with its groups in alphabetical order. The viewer now orders the two halves for display only (no call or data file changes): 1. Cross parents: where the parent's phase lanes (phase/atlas_phase_<res>.json) are A/B with confidence a/l/m and the pair is one clade-A group (lupulus, European lineages) plus one clade-B group (lupuloides, neomexicanus, American unresolved), h1's group is drawn on top. 2. Everything else and the fallback (AA/BB lanes, unlinked u, no call, same-clade pairs, cordifolius): each group keeps the half it held in the previous called heterozygous bin of the chromosome (greedy left to right, deterministic).

Hover text gives the unordered pair; the legend notes the display ordering.

python3 hop_atlas/tools/patch_halves_phase.py $STG     # lineage.js /*hphase:1..6*/; anchors asserted exactly once; idempotent; --check

Re-apply after a re-copy. As with /*cordi*/, the hooks live in the atlas copy of lineage.js: re-run the patch whenever atlas_layers.py patch re-copies it. Order relative to patch_cordifolius_level.py does not matter (no shared anchors).

8d. Level "WGS subspecies (variants)" (Lw, /*wgsv*/, 2026-10-10)

What it is (PI request 2026-10-10). A haplotype-resolution option for WGS rows only, built from the genome-wide joint call of 30 WGS samples (/data/hop_data/wgs_subspecies_variants/joint/, record sync/2026-10-10/wgs_subspecies_variants/). Five reference groups built from WGS plants only (lupulus, lupuloides, neomexicanus, pubescens, cordifolius) + U; the CALL is the coarse pair state lupulus / American / cordifolius / U, and the most likely American subspecies is an unvalidated, faded guess shown only with "fill" ticked. Rows outside the joint call are pale grey ("WGS-only level"). Nothing in lineage/, subev/ or subevc/ changes.

STG=/data/hop_data/lineage_refinement/stages/<name>/introgression_atlas
ln -sfn /data2/hop_data/wgs_subspecies_variants/viewer_wgsv_<stamp> $STG/wgsv
python3 hop_atlas/tools/patch_wgsv_level.py $STG       # re-apply after any re-copy of lineage.js / index.html

Rebuild after re-scoring (b03) or a new call rule: b06_export.py <model dir> <NEW dir>, re-point wgsv/, deploy. Codes are Greek letters (capitals = called coarse states, small = guesses), so they cannot collide with the ASCII codes of the other levels. Hook gotcha (2026-10-10): a hook marker inside a template literal is rendered as text (the first deploy showed "/wgsv:9/" in the legend); put markers inside a ${} expression. Also: never write /*marker*/ inside a JS block comment — it closes the comment.

8e. References and per-reference calls (/*frames*/, 2026-10-13)

What it is (PI request 2026-10-13). "Deprecate the old 'liftover' bins ... When we switch references, only display plants which have been aligned against that reference ... don't display references that have no bins/alignments/data." - The "Display coordinates" control is now Reference (?ref=<key>, default 21110M). It lists only references with calls on them, grouped by assembly (21110M · Saaz · 10-12 (HL1012) · Apollo · Cascade). Today 6: 21110M (served calls, all cohorts), Saaz hap1, 10-12 hap1, Apollo hap1, Apollo hap2, Cascade (126 Zenith x 21058M plants each: 124 progeny + both parents; Z_034/035/041/057 have no 09-25 BAMs). - Removed: the "Universal (consensus uMb)" option, the 14 lifted display frames, their drawing paths (per-frame placement of native bins, uMb axes, universal-frame branches) and the "Liftover coverage per assembly" table. The page no longer reads lifted geometry, so it was stripped from the served data (atlas_native_<res>.json per-bin g; atlas_data.json len / utile / notes.frame of every frame except 21110M, ulen, coverage). The universal bins and their 21110M placement stay: they carry the 21110M-only evidence tracks (haplotype rows, AIM, blocks, F segments, depth-class boundaries, terminal zones), and their hover tables still list the bin's coordinates in each assembly. - On 21110M the view is unchanged (canvas byte-identical to the served 21110M frame before the change, 5 Mb and 500 kb). - On any other reference frames.js swaps the viewer's data for that reference's calls: only the plants with calls on it, in its own calling bins and chromosome lengths. Sections, presets, Datasets drawer ("no calls on "), filters, sample search (": no calls on . It has calls on: ...") and ?sample= follow it. Layers without per-reference data are disabled (greyed, one-line note "available on 21110M only") and never drawn from 21110M data: haplotype rows, AIM, imputation no-call, breakpoint marks, phased parental lanes, switch-point guides, concordance badges, crossover ticks, transmission tracks, evidence tracks, terminal zones, sex-aware ChrX + ChrY, haplotype-resolution levels, sex / paternity badges, mixture-evidence and concordance filters, similarity sorts. Kept: the stacked summary and the per-panel mixture track (both counted from the drawn rows). - Hover on a non-21110M reference: call, sites with reads, posterior, A / B reads in that bin. Evidence detail: the per-frame run kept per-bin numbers only, so the panel says site-level detail exists on 21110M only. - ChrX on non-21110M references is drawn, labelled "ChrX (not sex-aware)": the per-frame caller uses one diploid model for every plant, and the 21110M re-call of ChrX agrees with the served sex-aware X in only 0.80 of 5,816 plant-bins (record §4a).

Registry frames/frames.json (17 entries): 21110M male composite; the 13 v5 panel haplotypes; Saaz hap1 (principal, PRJDB17944) and 10-12 hap1 (principal, PRJDB17947, + X/Y) as their own entries, because the per-frame saaz_h1 / hl1012_h1 references are NOT the panel Saaz / HL1012 sequences (4.25 M / 2.68 M substitutions; the panel ones are the pseudo-haploid deposits PRJDB17943 / PRJDB17946); Cascade (HopBase Dovetail). Panel Saaz / HL1012 (pseudo-haploid) are marked deprecated and are never offered (PI: hap1 / hap2 only for hop). Per entry: display name, group, hap, chromosome lengths in its own coordinates, bins per resolution (calling bins where calls exist, else nominal), cohorts with calls (samples, EM parameters, record), ChrX note. Per-frame key → registry: m21 → 21110M (the served calls are drawn; the re-call is not), apollo_h1 → Apollo_h1, apollo_h2 → Apollo_h2 (same sequences as the panel haplotypes, 0 substitutions), saaz_h1 → saaz_hap1, hl1012_h1 → hl1012_hap1, cascade → Cascade.

STG=/data/hop_data/lineage_refinement/stages/<name>/introgression_atlas
python3 hop_atlas/tools/build_frames.py $STG     # NEW data dir + $STG/frames symlink (reads panel lengths from atlas_data.json,
                                                 # or from the previous frames.json once the data were stripped)
python3 hop_atlas/tools/patch_frames.py $STG     # hooks + frames.js + strip lifted geometry (re-apply after any re-copy of index.html)

Adding a cohort's per-frame calls: run the per-frame program for the cohort (annevo_eval/methods/per_frame_ancestry/README.md "Extending to other cohorts"; output <work>/frames/<frame>/calls/<cohort>_<res>_calls.tsv.gz + <cohort>_params.json), add the cohort to COHORTS in build_frames.py (atlas cohort key, per-frame cohort name, sample-name → atlas key rule, record), re-run both tools on a stage, deploy. A reference gets offered as soon as one sample has calls on it. Adding a reference: an entry in REFS (key, display, group, hap, pf = per-frame frame key). A re-export that rewrites atlas_native_* / atlas_data.json brings the lifted geometry back (harmless; re-run patch_frames.py to strip it). The test suites that read lifted geometry (e.g. xytruth.py) need updating before they are next run.

9. Change log

• 2026-10-13: overview, patch-marker table and "How haplotypes are assigned" added at the top; §7 status corrected; self-contained zip packager (components/BUNDLE.md).
• 2026-10-13: references with per-reference calls; liftover frames removed (§8e), deployed 16:58.
• 2026-10-10: level "WGS subspecies (variants)" (§8d), deployed 17:49.
• 2026-10-09: display order of tile halves (§8c), deployed 22:19.
• 2026-10-09: experimental "Subspecies + cordifolius" level (§8b), deployed 21:42.
• 2026-10-09: subspecies / lineage evidence in the Evidence-detail hover boxes (§8), deployed 18:21.
• 2026-10-08: evidence pages + graded paternity badge + "See evidence" popover (§7), deployed 23:47.
• 2026-10-08: Zhang 2017 opt-in dataset, Datasets drawer and canvas guard (/*zhfold*/), deployed 23:46.
• 2026-10-08: document created; atlas_layers.py added; pedigree lookup deployed to :8900; layer scripts repointed from the session worktree to the main checkout's sync/ copies.

Display label of the Havill cross (2026-10-12). It is shown as "Zenith x 21058M"; the cohort key stays havill. Upstream exporters (m07/u14/o07-style scripts) still write "Havill cross (Zenith x 21058M)", so run python3 hop_atlas/tools/relabel_havill.py <stage>/introgression_atlas after any re-export, before deploying.