Hop atlas home · Overview · Changelog · Architecture · Adding_Data · Introgression · Ribbon · Dosage · Phylogeny · Bundle · Ingest · Ingest_Gbs · Ingest_Wgs · Ingest_Grasdi · Ingest_Rnaseq · Ingest_Assembly

Rendered from hop_atlas/components/RIBBON.md (2026-10-13)

Ribbon (structural) atlas: how it is built and how to update it

Status: living document, started 2026-10-08. The atlas and its products are tentative. Served: scratch_plots/structural_atlas, a symlink to sync/resources/ribbon_plots/structural_atlas/, on port 8900. - index.html: within-chromosome ribbons, reorderable rows. - crosschrom.html: chromosome-to-chromosome, circos and linear genome-wide views.

Both pages are self-contained: all geometry is embedded and nothing is fetched at run time. Tool: hop_atlas/tools/atlas_deploy.py (profile ribbon), described in §5.

Short answer: can a new assembly go into the ribbon atlas by following written instructions?

Partly. - Generic and runnable: rebuilding and verifying the atlas from atlas-ready inputs (§2, §5), and adding an assembly once its inputs exist (§3: one manifest entry, then rebuild). - Cohort-specific: producing those per-assembly inputs is documented only through the Cascade example (/data/hop_data/cascade_dovetail_atlas/report/PIPELINE.sh, "documentation of what was run, paths hardcoded"). - Not reproducible as written: the original 13-haplotype alignment depends on /workspace/joint_paintings_work/, which no longer exists. - Not available for new rows: repeat, ancestry and F-marker tracks have no documented recipe for new assemblies.

1. What is shown, and from what

• Rows (14):
• the legacy 13: m21_h1/h2 (21110M), apollo_h1/h2, saaz_h1/h2, hl1012_h1/h2, drHumLupu, MN1421_h1/h2, MN586_h1/h2;
• plus cascade_hopbase, which is tentative: an uncollapsed diploid that should be used with caution.
• Tracks: ribbons at 5 / 20 / 100 kb, 11 repeat families, NA/EU ancestry tiles, F markers, candidate loci (tentative), a mirror control, and in crosschrom a Cascade contig-junction track.
• Coordinates: Akagi numbering, with every assembly oriented drHumLupu-forward (Chr01–05 use x → L − x). The project rule in CLAUDE.md is to orient by dominant-strand voting before plotting.

Similarity view (similarity.html, added 2026-10-08): "which haplotypes have the most similar arrangement of chromosome X?" - Views: ranking (one haplotype + chromosome), one vs all chromosomes (grid), and matrix (all pairs on one chromosome); links work via #view=..&hap=..&chrom=... - Data: no new computation. It reads pair_distances_v2.tsv, the arrangement distance build_ribbon_v2.py already uses to order rows: 5 Mb backbone windows, coverage-weighted |Δstrand|/2 + 4·|Δposition offset|. It measures arrangement, not sequence identity. - Built by hop_atlas/tools/build_ribbon_similarity.py and tested by hop_atlas/tools/verify_ribbon_similarity.py: 8 rankings, the grid order and means, matrix counts and URL links against an independent reading of the TSV, with a shuffled-order must-fail. Both are part of atlas_deploy.py build/test ribbon. - Background: it replaces the unmerged tools/chr_similarity_explorer (branch worktree-saaz293-hsr-cascade-panel), which used a coarser ~1 Mb window join and also scored sequence sharing and identity. Sequence-based similarity would need computing from the existing PAFs; it is not in this view.

2. Build order (all in annevo_eval/scripts/)

• Staging: every step honours ATLAS_OUT (served files) and ATLAS_GEOM_OUT (geometry). That is how atlas_deploy.py build ribbon builds into a stage without touching the served copy.
• Recipes: the dated recipes are structural_atlas/PIPELINE*.sh (V2, CANDIDATES, CROSSCHROM, GENOMEWIDE, JUNCTIONS, CASCADE_ROW). They are documentation, not drivers.

3. Add a new assembly

1. Produce the atlas-ready inputs with the atlas recipe. Follow /data/hop_data/cascade_dovetail_atlas/report/PIPELINE.sh and its scripts/.
2. Chunk the assembly into 1 Mb pieces and align them to male_ref.fa with minimap2 -cx asm20 --cs -I 8G --secondary=no.
3. Run a reproducibility re-alignment of a chunk subset; the PAF lines must be byte-identical.
4. Orient by dominant-strand vote against drHumLupu (direct_vote.py).
5. Write the five inputs: compact/<hap>.tsv.gz, crosschrom/<hap>.json.gz, meta/<hap>_orientation_and_inversions.tsv, meta/<hap>_panel_spec.tsv and meta/<hap>_alignment_loss_audit.tsv.
6. Write a producer manifest with the sha256 of each file.
7. Those scripts are Cascade-specific (paths hard-coded), so copy and edit them for the new assembly.
8. Append an entry to annevo_eval/scripts/atlas_extra_assemblies.json: hap, label, short, alias, color, the five input paths, the producer manifest and a caveat.
9. Rebuild into a stage and test it (§5).
10. Missing tracks: the new row gets ribbons and cross-chromosome links. It gets no repeat, ancestry or F-marker track, because there is no documented recipe for those.

Changing the legacy 13 means editing several hard-coded lists: - HAPS / LABEL / COL in build_structural_ribbon_atlas.py; - NAMES in audit_ribbon_alignments.py; - ALIAS in build_ribbon_v2.py; - the haplotype lists in run_parse.sh, build_tables.py and run_align_all.sh.

Re-running the legacy alignment also needs /workspace/joint_paintings_work/{chunk_stream.py,keep/*.keep,ref/male_ref.asm20.mmi}, which is missing. It would have to be rebuilt from the Cascade scripts, which carry equivalent chunking (extract_chunks.py, subset_chunks.py).

4. Tests

Tests run only when the PI asks (PI, 2026-10-08). They are kept and documented, but not run after every change.

• How they run: the browser checks open the pages via file:// and need the chromelibs LD_LIBRARY_PATH / FONTCONFIG_FILE, which the tool sets.
• Results: they write *_validation.json next to the pages.
• Result on an unchanged stage of the served atlas, 2026-10-08: all six verifiers PASS (/data/hop_data/atlas_stages/ribbon/verify_2026-10-08/checks.txt).
• Must-fail: with one ribbon segment pushed past its chromosome's end in the stage geometry, verify_ribbon_v2_data fails. The geometry was then restored.
• Deploy dry run: it listed only the 10 re-rendered preview PNGs; the validation JSONs were byte-identical.

5. Stage → build → test → deploy

T=hop_atlas/tools/atlas_deploy.py
python3 $T stage ribbon --name my_change      # /data/hop_data/atlas_stages/ribbon/my_change/{structural_atlas,display_geometry.json}
STG=/data/hop_data/atlas_stages/ribbon/my_change
python3 $T build ribbon $STG                  # steps 1-5 of §2 into the stage (ATLAS_OUT / ATLAS_GEOM_OUT set)
python3 $T test  ribbon $STG                  # the six verifiers -> $STG/checks.txt
python3 $T deploy ribbon $STG --dry-run       # lists files; for text files, lines the served copy would LOSE
python3 $T deploy ribbon $STG                 # refuses if the served copy or the held geometry changed since `stage`

• What deploy does: it copies changed files into sync/resources/ribbon_plots/structural_atlas/ and the stage geometry to /data/hop_data/structural_ribbon_v2/display_geometry.json. Every replaced file is backed up as *.bak_<stamp>_<stage>, and nothing is deleted.
• History: past deploys used APPLY_*.sh scripts that lived only in session worktrees; this tool replaces them.
• Approval: changes to what is served need PI approval.

6. Known gaps

• Not in git: the ribbon scripts in annevo_eval/scripts/ are untracked.
• Incomplete provenance copy: annevo_eval/data_inventory/provenance_scripts/structural_ribbon_atlas_v2_2026_10_01/ lacks atlas_extra_assemblies.* and the junction modules.
• Stale records: the catalog entry structural_ribbon_atlas_v2_2026_10_01 and sync/resources/README.md predate the Cascade row, the candidate loci and the junctions.
• Missing alignment inputs: the legacy alignment inputs (/workspace/joint_paintings_work) are gone (§3).
• No recipe for new rows' tracks: repeat, ancestry and F-marker tracks.