Hop atlas home · Overview · Changelog · Architecture · Adding_Data · Introgression · Ribbon · Dosage · Phylogeny · Bundle · Ingest · Ingest_Gbs · Ingest_Wgs · Ingest_Grasdi · Ingest_Rnaseq · Ingest_Assembly

Rendered from hop_atlas/ARCHITECTURE.md (2026-10-13)

Hop atlas web app: architecture assessment and roadmap

Written 2026-10-08, prompted by "the atlases feel laggy; is this quick and dirty, or something we can grow, eventually into a public resource?". These are measurements and engineering notes, not scientific claims.

What it is today

Fully static. - Server: one Python ThreadingHTTPServer on :8900 (server/nocache_server.py) serves files from scratch_plots/. There is no backend, database or API. - Introgression atlas: - One hand-written page (index.html, about 113 kB) draws everything on a single canvas. - Data is about 30 JSON files per resolution, written by dated, cohort-specific scripts. - The haplotype-resolution and pedigree layers attach through 36 exact-once string patches (tools/atlas_layers.py patch). - Large detail data lives on /data, reached through symlinks. - Ribbon atlas: two generated, self-contained pages (16 MB and 28 MB) with all geometry embedded, plus the small similarity page. - Dosage atlas: a generated page plus gzip + base64 JavaScript data assets (about 290 MB in total).

Strengths to keep: - Every product is labelled tentative and has provenance. - Each component has browser test suites with must-fail checks. - Changes go through stage → test → deploy (tools/atlas_deploy.py) with drift checks and backups. - The data is static, so it could be hosted almost anywhere.

Measured load (introgression atlas, 2026-10-08, from this machine, so network time is near zero)

Why it feels slow: 1. Every control change redraws all rows on the main thread, including rows far off screen. That costs about 2 s per change at 5 Mb and more at finer resolutions, and it is the lag you feel. 2. Nothing is compressed or cached. Each visit re-downloads 7.8 MB, or about 36 MB at 500 kb. On a remote or VPN link that dominates. 3. Whole-genome JSON is loaded up front, not by chromosome or region. 4. The ribbon pages embed 16–28 MB each, parsed in full on every open.

Verdict

It is a disciplined prototype: well tested and documented, but grown feature by feature. It is not quick and dirty, but it is not built to scale either. - Can it keep growing? Yes, but each new cohort or layer adds more JSON and more string patches, so load time and fragility grow too. - As a public resource? Not as it stands. It needs the steps below first, and none of them requires a backend: the data is static and can be served from static hosting or a CDN.

Roadmap

Quick wins (hours; no redesign; behaviour unchanged): 1. Compression + caching. Serve with gzip/brotli and cache-friendly headers (versioned file names or ETags) instead of no-store. Expect 5–10× less transfer and near-instant reloads. Options: add both to the Python server, or put nginx/Caddy in front. 2. Draw only what is visible (required before the 4,000+ collection; see the opt-in design below). Use a viewport-sized canvas and draw only the rows on screen; redraw cost then scales with the screen, not the 2,372 rows. This is the biggest fix for the lag. 3. Debounce redraws and move per-bin aggregation (frequency tracks, lineage strings) to a Web Worker.

Medium (days to weeks): 4. One data build. A cohort manifest (accessions, assay, parents, roles) plus parameterised stages that regenerate every served file. That replaces the dated per-cohort scripts and the hand patches to index.html, and is the same step that closes the SRA → atlas ingestion gap in components/*.md. 5. Data by chromosome and resolution. Binary/columnar files (typed arrays or Arrow/Parquet), fetched on demand instead of genome-wide JSON. Ribbon geometry gets the same treatment, out of the HTML. 6. One front-end shell. Shared navigation, sample identity and URL state across the three components; front-end code as modules rather than patches. 7. Versioned data releases. releases/<date>/ with a manifest and checksums, and the viewer pinned to a release.

Public resource (beyond that): 8. Data rights. Check the release rights of every input, project by project. In particular, the CLAUDE.md note says the phased Apollo FASTA is embargoed and our phased Apollo is a DIY assembly; also check collaborator and unpublished material. Strip internal paths and notes. 9. Hosting. Static hosting (object store + CDN) with a release DOI. It needs a citation and provenance page, a tentative-status policy, browser and accessibility checks, and a light privacy-free analytics or feedback channel if wanted.

Design: opt-in datasets at 6,000+ samples (2026-10-08; drawer, lazy opt-in loading and canvas guard built the same day, the rest not built; see "Progress" below)

Why now. A collection of 4,000+ samples, mostly F1s at lower resolution, is coming. Many of them are relatives of OB79, the wild-neomexicanus-derived Wye male. - Today: 2,277 sample rows from 11 cohorts, all loaded on every visit. The largest cohorts are OSU 826 and Saaz 257. - The 5 Mb samples file is 1.9 MB; the 500 kb one is 13.5 MB. - Rows are drawn on one canvas 29,187 px tall. - At about 6,300 rows: - transfer about triples, with each redraw time; - the canvas would need about 77,000 px, beyond the browser's per-side canvas limit (about 32,767 px in Chrome and Firefox).

So "load everything, draw everything" stops working, not just slows down. The user has to choose what to see.

Pattern: a small index loaded always, data loaded on opt-in, rows drawn only when visible, families folded, presets as saved queries.

1. Dataset index (always loaded, small).
2. One entry per dataset (cohort):
   • key, title, data type (GBS / WGS / GRAS-Di / RNA-seq / DArT / genotype-only);
   • design (cross, panel or family collection), number of samples and families;
   • native resolution and coordinate frame, status, size per resolution;
   • whether it is on by default.
3. A sample directory for every sample in every dataset, loaded or not: key, display name, dataset, family (cross), mother/father, pedigree node, sex, flags. Estimated 0.5–1 MB at 7,000 samples, gzip-compressible.
4. Why the directory: search, the pedigree lookup and presets can then find samples in datasets that are not loaded.
5. Generation: from the cohort manifests (ingest/), so the ingest pipeline writes it.
6. Per-dataset data files loaded on opt-in.
7. Every served data file is split by dataset × resolution (later × chromosome), and a dataset's files are fetched only when it is switched on.
8. They are served gzip-compressed under versioned names (roadmap 1, 5).
9. Today's files mix all cohorts and have to be split.
10. A "Datasets" drawer.
11. Layout: datasets grouped by kind (crosses with reads, panels, genotype-only, large family collections), each with a checkbox, sample and family counts, download size and status.
12. State: the choice lives in the URL (?ds=havill,cascade,…), so links reproduce views, and is remembered per browser.
13. Search: searches the directory and offers to load: "12 matches in (not loaded): load?".
14. Default: a curated set, so a first visit stays fast. The new large collection is off by default.
15. Families as folded sections (Dataset → family → plants).
16. A folded family is one summary row: per-bin state frequencies, the mechanism the haplotype-resolution views already use for group frequency tracks. Clicking it unfolds the plants.
17. 4,000 F1s in ~180 families then open as ~180 rows. This, more than the drawer, makes the collection readable.
18. Draw only visible rows (roadmap 2; now required, not optional). The canvas is viewport-sized, rows are virtualised, and aggregation runs in a Web Worker. Redraw cost then depends on the screen, not the row count.
19. Presets as saved queries.
20. What a preset holds: {datasets, selector, sort, sections}. Choosing one switches on only the datasets it needs.
21. Selectors:
    • pedigree relation: relatives of X, up N / down M generations, both parents;
    • dataset or family list;
    • flag or tag;
    • explicit sample list.
22. Relation to the pedigree lookup: the Chinook lookup already does the pedigree walk; this generalises it from "draw the samples you have loaded" to "find and load the datasets that hold relatives".
23. OB79 preset: "relatives of OB79".
    • In our pedigree OB79 has 38 named descendants over 8 generations, including Target (Wye Target), Admiral, Pilgrim, Azacca, First Gold, Herald, Phoenix, Pioneer, Yeoman, Toyomidori and Exp-07270.
    • Sections by relation: offspring, grand-offspring, and so on.
    • Families count as relatives when a recorded parent is an OB79 descendant. Families with an unrecorded parent are labelled "relation via mother only".
    • If the collection is Zhang 2017, fathers are not recorded and 1,488 of 4,574 samples have no recorded mother. By mother only, about 189 samples are OB79 relatives: Wye Target 87, Exp-07270 95, Eastern Gold 4, Kitamidori 3.
    • Relatives found by genotype (kinship) would be a separate, labelled selector, never mixed silently with pedigree relatives.
24. Each dataset is drawn at its own native resolution. A coarser dataset shows as wider tiles; nothing is upsampled. It can only be shown on the 21110M frame once its markers are placed there. The Zhang 2017 HapMap's pseudo-chromosome coordinates are arbitrary, so it needs its markers placed via the scaffold coordinates first.

Build order (each step is usable on its own; ties to roadmap 1, 2, 4, 5): - (a) index + sample directory, and per-dataset file split, from the cohort manifests; - (b) Datasets drawer + lazy loading + URL state; - (c) visible-row drawing; - (d) folded families; - (e) presets as queries, including OB79; - (f) ingest the new collection as one new dataset (one experiment = one cohort; see ingest/README.md D1).

Steps (a)–(c) are needed before the new collection can be shown at all. Compression (roadmap 1) is a same-day win alongside.

Open questions for the PI: - which datasets are on by default; - is the incoming collection the Zhang 2017 HapMap or a different set; - should the OB79 preset include genotype-inferred relatives, as a separate section.

Progress since this assessment (updated 2026-10-13)

Still to do: compression + caching (roadmap 1), visible-row drawing (2; required before thousands more rows are on by default), debounced redraws / Web Worker (3), the base-atlas generator (4), data by chromosome in binary form (5), modular front end (6; the page is still grown by anchored patches, now about 20 markers), folded families (design step d), pedigree-relation presets (e), the per-dataset split of the default files (a), and everything under "Public resource". The load figures above were measured before the Zhang, Carey and Reference changes; the default view now has 2,299 sample rows on a 29,551 px canvas (canvas probe after Carey 2026, 2026-10-09), close to the limit, and the timings need re-measuring with probe_load.py when the PI asks.

Measurement script

hop_atlas/tools/probe_load.py (Playwright: bytes per load, time to ready, redraw time, resolution switch). Re-run it after each roadmap step.