Hop atlas home · Overview · Changelog · Architecture · Adding_Data · Introgression · Ribbon · Dosage · Phylogeny · Bundle · Ingest · Ingest_Gbs · Ingest_Wgs · Ingest_Grasdi · Ingest_Rnaseq · Ingest_Assembly

Rendered from hop_atlas/README.md (2026-10-13)

Hop atlas web app

One web app on port 8900, with four components. Landing page http://10.112.218.205:8900/ (same page at /hop_atlas/). Every page carries a shared navigation bar (www/nav.js; Home · Introgression · Badges · Ribbons · Cross-chromosome · Similarity · Dosage · Sex panel · Phylogeny · Docs). Current as of 2026-10-13.

Self-contained copy (zip): tools/package_atlas_bundle.py packages the introgression atlas, its Badges pages, the Phylogeny view and the docs into one zip with a generated CLAUDE.md (set-up / tear-down, design, build record); guide components/BUNDLE.md. Bundles live in /data2/hop_data/hop_atlas_export/.

Ingesting a new dataset from SRA into any component is not an end-to-end pipeline yet, but it is closer: cohort manifests exist for 9 cohorts (cohorts/), the manifest loader and jobs generator are built (ingest/pipeline/), and Carey 2026 was the first cohort brought in through a manifest plus numbered stage scripts (cohorts/carey2026/pipeline/, live 2026-10-09). Runbooks per data type (GBS ×3 kinds, WGS, GRAS-Di, RNA-seq, assemblies), reconstructed from what was actually run, plus the pipeline plan: ingest/README.md. Data type x atlas audit: ADDING_DATA.md. Each guide's "short answer" and gap list say which stages are scripted (alignment, rebuild, test, deploy) and which exist only as dated, cohort-specific records.

Folder layout

hop_atlas/
  README.md            this file: what the web app is, where each part lives, the rules
  CHANGELOG.md         dated record of what changed in the served app
  ARCHITECTURE.md      how it is built today, measured load, roadmap (performance, data build, public release) and what is done
  ADDING_DATA.md       data type x atlas audit: which workflows to add samples exist (documented / record-only / not possible)
  ingest/              one runbook per data type (GBS, WGS, GRASDI, RNASEQ, ASSEMBLY) + README (common skeleton, rules, PI decisions,
                       pipeline plan) + SAAZ_EXCLUSIONS.md + pipeline/ (manifest loader/validator, `jobs` generator, exclusions)
  cohorts/             one cohort manifest per cohort (cohort.yaml + samples.tsv): apollo, carey2026 (+ its pipeline/ c26_* scripts),
                       cascade, cordifolius, divpanel, havill, havill_blocks, saaz, wye
  components/          one guide per component (INTROGRESSION, RIBBON, DOSAGE, PHYLOGENY) + BUNDLE.md (the zip packager)
  evidence/            prose for the atlas evidence pages: BADGES.md (the badge page) + one .md per badge evidence page
                       (osu_paternity.md); tokens like {{FIG:oh_ranked}} become interactive figures (INTROGRESSION.md §7)
  server/              nocache_server.py + start_8900.sh: the :8900 server as it actually runs
  www/                 landing page + nav.js + rendered docs + files.html (served as :8900/hop_atlas/ via the symlink
                       scratch_plots/hop_atlas -> hop_atlas/www; scratch_plots/index.html -> www/index.html makes it the :8900 root page)
  tools/
    deploy, docs, nav  atlas_deploy.py (stage / build / test / deploy, all components) · atlas_layers.py (introgression add-on layers:
                       pedigree, lineage export, sex-aware X, viewer hooks) · render_docs.py (*.md -> www/docs/) · add_nav.py / check_nav.py
    introgression      every patch script is anchored (each anchor exactly once), idempotent and has --check; the marker it writes is in
      patches          brackets (full table: components/INTROGRESSION.md "Patch markers")
                       patch_datasets_guard.py [zhfold] + zhang_regroup.py [zhang2017]  Datasets drawer, opt-in Zhang 2017, canvas guard
                       patch_evbadge.py [evbadge] + build_paternity_badges.py + build_evidence_pages.py (+ evidence_assets/, evidence_common.py)
                       patch_sample_clear.py [sclear] · patch_url_sample.py [urlsample] · relabel_havill.py (Zenith x 21058M labels)
                       build_subspecies_evidence.py + patch_subspecies_evidence.py [subev] (+ subev_assets/)
                       patch_cordifolius_level.py [cordi] · patch_halves_phase.py [hphase]
                       patch_wgsv_level.py [wgsv] (+ wgsv_assets/) · patch_wgsv_fill_default.py [wfill]
                       build_frames.py + patch_frames.py [frames] (+ frames_assets/)  Reference selector, per-reference calls
    other components   build_phylogeny_view.py + phylogeny_view/ (Phylogeny) · build_ribbon_similarity.py / verify_ribbon_similarity.py
                       (ribbon) · relocate_dosage.py / check_dosage_http.py (dosage)
    packaging          package_atlas_bundle.py + bundle_assets/ (CLAUDE.template.md, serve.sh, stop.sh): self-contained zip (BUNDLE.html)
    probes and tests   probe_load.py (load / redraw timing) · probe_canvas.py (canvas paints, row count) · test_evidence_badges.py
                       (run only when the PI asks)

Data and build directories (not in git): - /data/hop_data/lineage_refinement/ (introgression layers; viewer_lineage* symlinked as introgression_atlas/lineage); - /data/hop_data/structural_ribbon_v2/ (ribbon geometry and intermediates); - /data/hop_data/dosage_explorer_v2/ (dosage builder stage and work); - /data/hop_data/hop_atlas/dosage_sources/ (the dosage folders moved out of sync/); - /data/hop_data/atlas_stages/ (stages made by atlas_deploy.py); - /data/hop_data/lineage_refinement/stages/<name>/ (introgression stages made by atlas_layers.py stage); - /data/hop_data/lineage_refinement/viewer_subspecies_evidence_* (subev/, subevc/), /data2/hop_data/wgs_subspecies_variants/viewer_wgsv_* (wgsv/), /data/hop_data/per_frame_ancestry/viewer_frames_* (frames/), /data/hop_data/hop_atlas/phylogeny/ (Phylogeny); - /data2/hop_data/hop_atlas_export/ (self-contained bundles and their zips; components/BUNDLE.md).

Rules for changing what is served

1. Stage → build → deploy with hop_atlas/tools/atlas_deploy.py / atlas_layers.py (no review server or sign-off step; PI, 2026-10-09: this is an internal development view, and short breakage while developing is acceptable). Other sessions edit these atlases too: on 2026-10-07 a cohort was added while another change was staged. stage copies the current served copy, and deploy refuses if it changed since. Deploy shows which lines would be lost, backs up every replaced file (*.bak_<stamp>_<stage>) and deletes nothing.
2. Deploy live when the change is built if the PI asked for it. Note every served change in CHANGELOG.md; if something breaks, the *.bak_<stamp>_<stage> backups restore it.
3. No testing by default (PI, 2026-10-08 / 10-09). The test suites, probes and review servers are kept for when the PI asks; they are not run after a change. Keep the deploy tool's drift refusal: it is not a test, it stops one session overwriting another's served edits.
4. Work in the main checkout, not a worktree. Commit only your own paths: main usually carries other sessions' uncommitted edits.
5. Keep served pages out of sync/. sync/ is a dated handoff log, and its folders get archived.
6. Re-apply the patches after a re-copy. The introgression page and lineage.js carry anchored patches (markers like /*frames*/). A cohort script that rewrites index.html, or atlas_layers.py patch re-copying lineage.js, drops the later hooks: re-run the patch scripts listed in components/INTROGRESSION.md ("Patch markers") and relabel_havill.py on the stage before deploying.

Server

:8900 is hop_atlas/server/nocache_server.py 8900, a static file server rooted at scratch_plots/ that sends no-cache headers and follows symlinks. Running since 2026-10-03 (scratch_plots/.nocache_server.py, identical file). To (re)start it: hop_atlas/server/start_8900.sh. Note that scratch_plots/serve.sh starts a plain, caching http.server instead. - Retired: :8906 (dosage, served all of sync/), stopped 2026-10-08. - Retired: the lineage review copies :8902 (v1) and :8905 (v2), stopped 2026-10-08; their stage folders were removed (unused).

Running the browser tests

Since 2026-10-08 the container's system copy of Chromium's libraries (libnspr4, nss3) is gone, so every Playwright test needs: export LD_LIBRARY_PATH=/data/hop_data/hop_atlas/browser-libs/x86_64-linux-gnu:/opt/conda/lib:$LD_LIBRARY_PATH FONTCONFIG_PATH=/opt/conda/etc/fonts FONTCONFIG_FILE=/opt/conda/etc/fonts/fonts.conf. The libraries are a durable copy (5.7 MB) of the ribbon pipeline's /tmp/ribbon-browser-libs extraction (see annevo_eval/data_inventory/provenance_scripts/structural_ribbon_atlas_v2_2026_10_01/PIPELINE_CROSSCHROM.sh).

Open items

• Ingestion: runbooks written 2026-10-08 (ingest/); manifests + jobs built (increment 1); call + export (the base-atlas generator) not built. PI decisions D2 and D4–D7 in ingest/README.md are still open.
• Introgression:
• per-reference calls exist for one cohort (Zenith x 21058M, 126 plants) on 5 references; other cohorts are being added (§8e of the guide). On a non-21110M reference all layers except the base calls are disabled;
• subspecies and lineage levels cover the cohorts with reads only (not Pilgrim, not Zhang 2017); the WGS-variants level covers the 29 WGS rows only;
• the canvas still draws every row (visible-row drawing, folded families and server-side compression are not built; ARCHITECTURE.md);
• the test suites that read lifted geometry (e.g. xytruth.py) need updating before they are next run;
• the 37 non-exact pedigree name matches await PI review.
• Ribbon:
• the legacy alignment can't be re-run (/workspace/joint_paintings_work is missing);
• the build scripts are not in git;
• the similarity view ranks by arrangement only; sequence-based similarity (shared bp, identity) would need computing from the PAFs.
• Dosage: cohort and reference lists are duplicated across about five builder files; the generic hop_adosage package (merged in 029bf81) is not yet used by the served build.
• Phylogeny: static output of the 2026-10-10 run; no pipeline of its own.
• Bundle: --include structural_atlas,dosage_atlas is supported but has not been run.