Hop atlas home · Overview · Changelog · Architecture · Adding_Data · Introgression · Ribbon · Dosage · Phylogeny · Bundle · Ingest · Ingest_Gbs · Ingest_Wgs · Ingest_Grasdi · Ingest_Rnaseq · Ingest_Assembly
Rendered from hop_atlas/ingest/RNASEQ.md (2026-10-13)
Runbook: ingesting RNA-seq samples (SRA/ENA → alignment → NA/EU calls → introgression atlas :8900)
Status: reconstructed 2026-10-08 from project history (scripts, logs, READMEs). Nothing was re-run to write it.
Is it generic? No. Only the Wye RNA lane (wyerna, PRJNA694508) has gone all the way to the atlas. Every stage
below is a copy-and-edit of that cohort's dated scripts unless it is marked GENERIC.
hop_atlas/ADDING_DATA.md rates RNA-seq as "REC" (record only), and this runbook does not change that rating.
Conventions: (inferred) marks a step that no record shows being run. [manual] marks a step that was done by hand, with no script.
Abbreviations used throughout:
| Abbreviation |
Path |
W1 |
sync/2026-10-01/wye_rnaseq_PRJNA694508/ (download → align → QC → joint calls → identity gate) |
W2 |
sync/2026-10-02/rnaseq_wye_target_x_2_1/ (coverage, StringTie, phasing, fusion, ASE; downstream research) |
OA |
sync/2026-10-02/osu_introgression_atlas/ (o13–o22; the live copies are in /data/hop_data/osu_introgression_atlas/scripts/) |
MR |
sync/2026-09-30/multires_clade/ (m01/m03/m04, the base-atlas counting and calling that o13/o03/o04 copy) |
D |
/data/hop_data/wye_rnaseq_PRJNA694508/ (raw data, BAMs, logs). Archived to PBSS 2026-10-08 (s3://general-purpose/skothenhill/archive/PRJNA694508/), so it may be partially gone locally |
1. Scope
| Cohort |
Accession |
Samples |
Read layout / platform (measured) |
Reference(s) used |
Downstream use |
Reached the atlas? |
| Wye Target × BL2/1 leaf RNA-seq (Prentout et al. 2021 New Phytol 231:1599, doi:10.1111/nph.17456) |
PRJNA694508, SRR13528958–971 |
14: mother WT_fpa (= Wye Target WGS), father R2_1_mpa (= BL2/1 WGS), 12 F1 (131_*, 132_*). Sample name = library_name with /→_ |
PE 2×150 bp (the paper says 100 bp), unstranded (featureCounts test, W2/scripts/r60_annotation_and_strand.sh), poly-A, young leaf |
21110M male_ref.fa (served lane); also Apollo h1 and MN586 hap2 |
Introgression atlas wyerna lane (v3, 2026-10-03); coverage, StringTie, phasing, ASE, fusion (W2); Chr01×Chr05 write-up (sync/2026-10-06/wye_target_rnaseq_chr01_chr05/) |
yes |
| Apollo (hybrid) RNA-seq atlas |
PRJEB63995 (Carlsberg / Kale 2026), 71 of 109 runs on disk at /data/hop_data/Apollo_rnaseq_fastq |
multi-tissue |
PE 100 bp, HiSeq 2000 (HWI-ST565) although ENA says "NovaSeq 6000" (annevo_eval/docs/NA_EU_APOLLO_DATA_INVENTORY.md §caveats 1–2) |
none found |
BRAKER3 annotation evidence (catalog apollo_rnaseq_atlas) |
no; no alignment record found |
| ANNEVO comparator hop RNA (8 tissues) |
ERR11670097/125/127/130/133/137/148/154 |
8 tissues |
PE (annevo_eval/data/RNASEQ_PROVENANCE.md) |
drHumLupu1.1 (GCF_963169125.1); Cascade (HopBase) for BRAKER3 |
BRAKER3/MAKER comparator, rnaseq_arbitrate.py |
no. BAMs, index and FASTQ moved to trash_rootdisk/ on 2026-10-02 (annevo_eval/data/humulus/STAR_RNASEQ_BAMS.MOVED.txt; catalog drhumlupu_rnaseq) |
| Mimulus RNA (comparator) |
PRJNA1054563, PRJNA1049363 (+ nasutus/tilingii) |
7+ |
PE; HiSeq 2500 / NextSeq 2000 |
E. guttata GCA_051857295.1 |
BRAKER3 comparator |
n/a (not hop) |
Provenance conflict (unresolved). RNASEQ_PROVENANCE.md says the 8 ERR11670xxx runs come from PRJEB63995. Catalog
drhumlupu_rnaseq says "ENA SRA (drHumLupu / DToL)". Check ENA before reusing them.
2. Stages, in order
Stage 0. Design decisions to make before starting
- Reference frame. The atlas is in the 21110M
male_ref.fa frame (/data/hop_data/ref_21110M_male/male_ref.fa, md5 83e3f171…). The clade catalogue /data/hop_data/clade_ab_imputation/diag/ is defined in that frame. An RNA lane must be aligned to male_ref; extra references are optional.
- Parent identity. A new cohort needs WGS or GBS of the same plants for the identity gate (Stage 3b) and for the RNA-vs-WGS agreement check (Stage 6).
- Disk. Wye needed about 14 × 3 CRAMs plus indexes.
/data is tight, so outputs were written as CRAM because of the 100 GB floor (W1/README.md).
Stage 1. Download
|
|
| Scripts |
D/download_PRJNA694508.sh (documentation of commands), D/run_aria2.sh (resumable run actually used), D/make_manifest.py |
| Tool + flags |
ENA filereport curl 'https://www.ebi.ac.uk/ena/portal/api/filereport?accession=<PRJ>&result=read_run&fields=run_accession,library_name,fastq_ftp,fastq_md5,fastq_bytes,read_count,base_count&format=tsv', then aria2c -i aria2_input.txt -d fastq -j 4 -x 2 -s 2 -k 20M -c --check-integrity=true --max-tries=20 --retry-wait=30 --auto-file-renaming=false --allow-overwrite=false --file-allocation=none. aria2_input.txt has one block per file: URL / out= / checksum=md5=<ENA md5>. Its builder is not recorded [manual]. |
| I/O |
in: ENA filereport. out: D/fastq/<RUN>_{1,2}.fastq.gz, D/manifest_PRJNA694508.tsv (run, library_name, role, sex, file, bytes, md5, md5_ok, read_count), D/md5sum_computed.txt |
| QC gate |
make_manifest.py computes the md5 of every file independently and compares it to ENA fastq_md5. It exits 1 on any FAIL. Then peek at the headers to record the instrument and read length as observed (the loop is in download_PRJNA694508.sh). |
| Gotchas |
ENA refused connections at -j10 -x8; the run was restarted at -j4 -x2 (run_aria2.sh comment). ENA/SRA instrument and read-length labels are unreliable: Wye is 150 bp, not the paper's 100 bp, and Apollo is HiSeq, not NovaSeq (memory sequencing-platform-labels-unreliable). Verify duplicates by checksum, not size (memory verify-duplicates-by-checksum-not-size). On EOS, stage via PBSS and never re-download there (CLAUDE.md). |
| Generic? |
COPY-AND-EDIT. In make_manifest.py, edit D, the filereport name and the hard-coded role/sex rule (WT_fpa → mother, R2_1_mpa → father, else F1 with sex = last character of library_name). Alternatives seen: annevo_eval/scripts/download_rnaseq.sh (prefetch + fasterq-dump + pigz, manifest-driven, used for the comparator) and annevo_eval/scripts/dl_apollo_rnaseq.sh (wget over a URL list). |
After download, register the dataset in annevo_eval/data_inventory/catalog.yaml (Wye: entry PRJNA694508_rnaseq_fastq) and run refresh.sh. The Wye catalog entries were written by W2/scripts/r90_catalog_entries.py; the log is D/logs/refresh_catalog_694508.log.
PBSS mirror: aws s3 sync of fastq/ to s3://general-purpose/skothenhill/archive/PRJNA694508/fastq/. It ran inside W2/scripts/r30_stage_for_lepton.sh and was size-verified. A grep of archive_ledger.json for PRJNA694508 on 2026-10-08 found no entry, so if today's archive went through archive.sh, check the ledger.
Stage 2. Read QC
|
|
| Scripts |
None recorded. No fastp, FastQC or trimming was run on Wye RNA, and STAR soft-clips. |
| QC actually done |
md5 (Stage 1), plus post-alignment QC (Stage 3c). |
| Gotcha |
--read-files-command zcat is needed for gz FASTQ or STAR dies with "read ID should start with @ or >" (memory parabricks-rnaseq-alignment). |
| Generic? |
(inferred) If wanted, cluster/lepton_linear/linear_job.sh has a DO_FASTP switch, but it is a DNA alignment job and has never been used for RNA. |
Stage 3a. STAR index
|
|
| Scripts |
W1/scripts/r01_star_index.sh (env ARMS, SPARSE); W2/scripts/r30_stage_for_lepton.sh (dense 21110M index for Lepton) |
| Tool + flags |
STAR 2.7.2a in quay.io/biocontainers/star:2.7.2a--0 (the version Parabricks 4.7.0 documents): STAR --runMode genomeGenerate --genomeFastaFiles <fa> --runThreadN 12 --genomeSAsparseD <1|2> --genomeSAindexNbases 14 --genomeChrBinNbits 18 --limitGenomeGenerateRAM 55000000000, docker --memory 60g. No GTF / no sjdb (1-pass, de-novo junctions). |
| Existing indexes |
/data/hop_data/star_index_21110M (sparseD 2, ~14 GB, CPU), star_index_21110M_sa1 (dense, ~25 GB; built by r30 for Lepton, also uploaded to PBSS PRJNA694508/star_index_21110M_sa1/), star_index_MN586{,_sa1}, star_index_apollo_h1 (sparse 2) |
| QC gate |
SAindex and SA non-empty (r01 skips existing ones). |
| Gotchas |
GPU STAR in Parabricks 4.7.0 refuses sparse SA indexes (gSAsparseD != 1, was 2), so a dense (--genomeSAsparseD 1) index is required for pbrun. It takes about 15 min and about 45 GB RSS (W1/README.md "Is Parabricks useful here?" item 1). Parabricks does not build indexes. Docker is the host daemon: -v must use host paths for /workspace (memory parabricks-rnaseq-alignment); /data maps 1:1. |
| Generic? |
COPY-AND-EDIT: the REF associative array and OUT. Reuse the existing 21110M indexes for an atlas lane. |
Stage 3b. Alignment
Two routes were used for Wye. The served atlas lane uses the CPU route on 21110M.
|
GPU: pbrun rna_fq2bam |
CPU: STAR + samtools (used for bam_21110M, the atlas input) |
| Script |
W1/scripts/r02_pbrun_rna_fq2bam.sh (+ r02w_gpu_wait_wrapper.sh retry loop); MN586 arm |
W1/scripts/r02c_cpu_star.sh; 21110M and apollo_h1 arms |
| Command |
pbrun rna_fq2bam --ref <fa> --genome-lib-dir <idx_sa1> --in-fq R1 R2 --read-files-command zcat --read-group-sm <sm> --read-group-lb <sm> --read-group-pl ILLUMINA --read-group-id-prefix <run> --out-sam-mapq-unique 60 --num-gpus 1 --num-streams-per-gpu 1 --memory-limit 16 --output-dir star_<arm>/<sm> --out-prefix <sm>. --out-duplicate-metrics bam_<arm>/<sm>.dupmetrics.txt --out-bam bam_<arm>/<sm>.cram, container nvcr.io/nvidia/clara/clara-parabricks:4.7.0-1, docker --memory 60g |
STAR --runThreadN 12 --genomeDir star_index_<arm> --readFilesIn R1 R2 --readFilesCommand zcat --outSAMtype BAM Unsorted --outStd BAM_Unsorted --outSAMmapqUnique 60 --outSAMattrRGline ID:<run>.1 SM:<sm> LB:<sm> PL:ILLUMINA PU:<run>.1 --outFileNamePrefix star_<arm>/<sm>/<sm>. piped to samtools fixmate -m → samtools sort -m 1500M → samtools markdup -s --reference <fa> -O cram (marks duplicates, does not remove them); docker --memory 34g |
| Defaults |
1-pass (--two-pass-mode None); duplicate marking ON (tool default) |
1-pass; duplicates marked |
| I/O |
D/fastq/ → D/bam_<arm>/<sm>.cram(.crai), D/star_<arm>/<sm>/<sm>.{Log.final.out,SJ.out.tab} |
same layout + <sm>.markdup_stats.txt |
| Timing (WT_fpa, 32.4 M pairs) |
9.0 min in total on 1 A6000 |
10 min mapping only (12 threads); 6.5–15 min per sample with sort and markdup (D/logs/r02c_cpu_walltimes.tsv) |
| Gotchas |
--num-gpus 2 under a 60 GB cap thrashes (each GPU worker holds a genome copy; pbrun auto-tunes to host RAM, not the cgroup), so use 1 GPU (~57 GB RSS). Libraries over about 50 M pairs with 2-pass OOM on 125 GB, so use --low-memory --memory-limit 100 (memory parabricks-rnaseq-alignment). Speed-up is only about 1.2× on mapping (W1/README.md). |
waits while /data < 105 GB or RAM < 45 GB (built-in guard) |
- Standing rule vs practice. CLAUDE.md says "all read alignment via GPU pbrun". The 21110M and apollo_h1 arms were run on CPU STAR as a coordinator decision ("option b": no disk for two more 25 GB dense indexes;
W1/README.md).
- Catalog
PRJNA694508_rnaseq_alignments says "pbrun rna_fq2bam … GPU STAR" for all three arms. That is wrong for bam_21110M/bam_apollo_h1: markdup_stats.txt exists and r02c_cpu_walltimes.tsv covers all 28.
- Fix the catalog text before relying on it.
- Chimeric / fusion run (optional, downstream only). Lepton
gpu.8xh100-sxm job hop-rnaseq-fusion-21110m, W2/scripts/r30–r34.
pbrun rna_fq2bam with --min-chim-segment 12 --min-chim-overhang 8 --max-chim-multimap 20 (catalog PRJNA694508_rnaseq_alignments).- Inputs and outputs go through PBSS
PRJNA694508/{fastq,star_index_21110M_sa1,fusion_21110M}.
- The exact pbrun option names are in
r31_lepton_fusion_21110M.py SCRIPT; read them there, not from the catalog's summary.
- ANNEVO / BRAKER route (annotation evidence, not atlas):
annevo_eval/scripts/align_rnaseq_parabricks.sh and cascade_braker3.sh.
pbrun rna_fq2bam --two-pass-mode Basic --num-gpus 2, index built with STAR 2.7.2a --genomeSAindexNbases 14 (hop) or 13 (Mimulus), --limitGenomeGenerateRAM 90e9.- Results:
annevo_eval/data/BAM_ALIGNMENT_RESULTS.md.
- Generic? COPY-AND-EDIT in r02/r02c:
BASE, MAN (manifest path and column 1 = run, column 2 = library_name), the REF map, the index path and the sample-name rule sm=${lib//\//_}.- The script assumes
<run>_1/_2.fastq.gz under $BASE/fastq.
Stage 3c. Alignment QC and identity gate
|
|
| Scripts |
W1/scripts/r03_bam_qc.sh + r03_qc_table.py → D/qc/rnaseq14_alignment_qc.tsv; W1/scripts/r05a_wgs_sites.sh + r05_parent_identity_gate.sh + r05_gate.py → D/calls/gate/parent_identity_gate.tsv; r06_sample_sanity.py → D/calls/rnaseq14_sample_sanity.tsv |
| Tool + flags |
samtools flagstat; samtools view -F 0xF04 -q 20 | cut -f3 | uniq -c (per-chromosome usable reads); STAR Log.final.out unique/multi/unmapped %. Gate: WGS parent sites bcftools view -m2 -M2 -v snps -i 'QUAL>=30 && FMT/DP[0]>=10 && FMT/DP[1]>=10 && GT[0]!="mis" && GT[1]!="mis', then RNA parents mpileup -T <sites> --ff UNMAP,SECONDARY,QCFAIL,DUP -q 20 -Q 20 --max-depth 2000 -a FORMAT/AD,FORMAT/DP | call -m -C alleles -T <sites>, compared at DP≥10 in both |
| QC gate (measured Wye) |
matching parent pairs: GT concordance 0.79–0.81, opposite-hom 0.003; swapped pairs 0.54–0.58 / 0.02 (W1/README.md). Unique mapping is highest on 21110M (mismatch about 0.4 %) (W2/README.md) |
| F1 identity / duplicates / mixtures |
RNA F1 vs the 142 GBS F1: sync/2026-10-01/chr5_inversion_survey/scripts/s28_rnaF1_vs_gbs_identity.py (opposite-hom rate; superseded by sync/2026-10-06/wye_target_rnaseq_chr01_chr05/scripts/w05_identity_sex_chimera.py, tables U1/U1b). Found: 132_82F = 132_81F (0 opp-hom / 112,716 sites); 131_70F mixture-like |
| Sex check |
samtools view -c -q 60 -F 0xF04 -T male_ref.fa <cram> ChrY:1-130000000 vs ChrX:1-230000000 (sync/2026-10-06/.../scripts/w05b_sex_msy.sh) |
| Generic? |
COPY-AND-EDIT. r05 hard-codes the MN586 arm, the WGS VCF /data/hop_data/wye_refscan/MN586_neomexicanus_HAP2/parents_MN586_neomexicanus_HAP2.vcf.gz and the parent CRAM names. r05 also calls r05_gate.py through a worktree path (.claude/worktrees/chr5-inversion-survey/...); repoint it to sync/. |
Stage 3d. Joint RNA VCF (not used by the atlas)
|
|
| Script |
W1/scripts/r04_bcftools_joint_call.sh (per chromosome, then concat) |
| Command |
bcftools mpileup -f <fa> -b bams.txt --ff UNMAP,SECONDARY,QCFAIL,DUP -q 20 -Q 20 --max-depth 2000 -a FORMAT/AD,FORMAT/DP | bcftools call -m -v → D/calls/<arm>.rnaseq14.vcf.gz (bcftools 1.20, annevo_eval/tools/bin/bcftools) |
| Gotcha (critical) |
Do NOT feed this VCF to the atlas. It is variants-only (-v). On the 21110M catalogue REF = EU (A) at 100 % of autosomal sites, so cohort-wide EU sites drop out and NAm/het fractions inflate on AA chromosomes (e.g. Target Chr04 0.40 vs 0.009). The first wyerna build (route V: clade_ab_imputation/scripts/07_extract_cohort.py wyerna <vcf> → 08_impute.py → o04) was superseded and never served (OA/README.md §5; sync/2026-10-03/wye_ancestry_method_validation/README.md; memory variants-only-site-selection-trap). The r40/r94/r107/r108 RNA ancestry numbers in W2 were retracted for the same reason (W2/README.md CORRECTION). |
| Also |
r04 hard-errors unless exactly 14 BAMs are present ([[ $n -eq 14 ]]), so edit that for another cohort size. RNA keeps DUP excluded: "this is RNA-seq, the GBS rule does not apply" (W1/README.md). |
Stage 4. Allele counts at the diagnostic catalogue sites (pileup)
|
|
| Script |
OA/scripts/o13_rna_pileup.sh (a copy of MR/scripts/m01_count_one.sh, WGS branch) |
| Command |
bcftools mpileup -f male_ref.fa -T /data/hop_data/clade_ab_imputation/diag/targets_all.tsv.gz --min-MQ 20 --min-BQ 20 --ff UNMAP,SECONDARY,QCFAIL,DUP -a FORMAT/AD -Ou <cram> | bcftools query -f '%CHROM\t%POS\t%REF\t%ALT[\t%AD]\n' | gzip -1, 5 samples in parallel (xargs -P 5) |
| I/O |
in D/bam_21110M/<sample>.cram; out /data/hop_data/osu_introgression_atlas/bamcounts/wyerna/<sample>.tsv.gz + .done + .err (about 0.3 GB in total) |
| QC gate |
both pipe stages rc 0 and > 1,000 output lines, or FAIL is written to .err |
| Gotchas |
Counts all covered catalogue sites, reference reads included; this is the fix for the variants-only trap. DUP is dropped because the CRAMs are duplicate-marked (WGS branch). Never use the GBS branch (keep DUP) on RNA. Conversely, never drop DUP on GBS (CLAUDE.md, ASSAYS.md). MQ20 keeps only STAR-unique reads (MAPQ 60). |
| Generic? |
COPY-AND-EDIT: the input glob /data/hop_data/wye_rnaseq_PRJNA694508/bam_21110M/*.cram and OD. m01_count_one.sh <cohort> <sample> <pbss_key> WGS is the generic form, but it streams BAMs from PBSS via presigned URL and writes to multires_clade/bamcounts/, which o03 does not read. |
Stage 5. Cohort matrix and NA/EU (clade A/B) calls
|
|
| Scripts |
OA/scripts/o03_build_bam_cohort.py (copy of MR m03), OA/scripts/o04_impute_multires.py (copy of m04) |
| Commands |
SAMPLES=/data/hop_data/osu_introgression_atlas/wyerna_samples.txt python3 o03_build_bam_cohort.py wyerna, then python3 o04_impute_multires.py wyerna_bam |
| Method |
o03: nA/nB = reads matching the catalogue A/B base (no REF/ALT assumption); keep sites with ≥ 1 read in ≥ 5 samples; thin to one site per 150 bp cell (max depth). o04: EM fitted once at 5 Mb on the cohort's own counts, held fixed at 0.5/1/2/5 Mb; tau 0.5; call = pmax ≥ 0.95 and ≥ 3 sites. |
| I/O |
wyerna_samples.txt (14 names, one per line, [manual]) + bamcounts/wyerna/ → /data/hop_data/multires_clade/cohorts/wyerna_bam.npz → multires_clade/calls/wyerna_bam_{500kb,1Mb,2Mb,5Mb}_calls.tsv.gz, imputation/wyerna_bam_params.json |
| QC gate |
o03 asserts a .done for every sample. Wye EM converged in 10 iterations (Δloglik < 1). Params e 0.0013, eta 0.0228, dA 0.071, dB 0.177, w 0.701; 253,491 sites (OA/README.md EM table). |
| Gotchas |
o03 hard-codes the pileup root OSU_BC=/data/hop_data/osu_introgression_atlas/bamcounts (only SAMPLES is an env var). With 14 samples the ≥ 5-sample site rule is strong. Whether new samples should refit the EM or reuse the old parameters is undocumented (ADDING_DATA.md). |
| Generic? |
COPY-AND-EDIT (OSU_BC, the cohort name). The cohort name must end _bam for o04 to take the BAM branch. |
Stage 6. Evaluation
|
|
| Script |
OA/scripts/o16_rna_eval.py |
| Outputs |
/data/hop_data/osu_introgression_atlas/tables/R1_wyerna_by_resolution.tsv (callable fraction, RNA parent vs same-plant WGS agreement, duplicate-pair agreement), R2_wyerna_parent_vs_wgs_by_chrom.tsv, R3_wyerna_parent_states_chr04_chr08.tsv |
| QC gate (Wye) |
callable 0.83 / 0.94 / 0.99 / 0.99 (500 kb → 5 Mb); RNA parents vs WGS (wye_bam) 0.976 → 0.998; 132_81F vs 132_82F ≥ 0.999. o15 reads R1, so o16 must run before o15. |
| Generic? |
COPY-AND-EDIT. These are hard-coded: PAIR = [('WT_fpa','WyeTarget'),('R2_1_mpa','BL2_1')], the WGS comparator wye_bam, the duplicate pair 132_81F/132_82F, and the Chr04/Chr08 focus. |
Stage 7. Atlas export (viewer JSON)
|
|
| Scripts |
OA/scripts/o15_wyerna_export.py (rows, transitions, similarity, reliability, index.html patch), OA/scripts/o17_detail_osu.py wyerna (per-sample detail files), OA/scripts/o18_y_hover.py (ChrY hover, all cohorts) |
| Commands |
ATLAS_DIR=<stage>/introgression_atlas python3 o15_wyerna_export.py; ATLAS_DIR=<stage>/introgression_atlas python3 o17_detail_osu.py wyerna |
| What o15 writes |
multires/atlas_{samples,transitions,similarity}_<res>.json, multires/atlas_reliability.json, index.html: PANEL_ORDER entry ['wyerna',['RNA parents','RNA F1s']], per-bin ns (n_sites) in 3 hover tips, preset ?preset=wye_rna. Data-layer label: "RNA (expressed-gene AD at diagnostic sites)", plus a caveat string. Checks → tables/O15_wyerna_export_checks.tsv (asserts all ok). |
| o17 writes |
/data/hop_data/universal_introgression_atlas/viewer_detail_mr/{sites,<res>}/wyerna__<sample>.json.gz (served as detail_mr) |
| Gotchas |
o15 backs up each file once to viewer_backup_pre_rna/ and every run restarts from those backups (idempotent). Re-running it today on a fresh stage would therefore overwrite later changes: the Pilgrim cohort, lineage and pedigree hooks were added after 10-03, and p05_atlas_export.py string-replaces the wyerna PANEL_ORDER entry. o15 asserts that cohort wyerna is not already present, so it cannot update an existing lane. o17 reads accuracy from M1/O1 tables; there is no wyerna truth plant. |
| Hard-coded per cohort |
COH, GROUPS, LAYER/CAVEAT text, NAM/NON carrier sets, the meta() row names/aliases/flags (parents, 131_70F mixture, 132_82F hidden), sample table path W2/tables/sample_table.tsv (written by W2/scripts/r20_datasheet.py) |
| Generic? |
COPY-AND-EDIT, high effort. There is no generator for the base viewer (INTROGRESSION_ATLAS_TECHNICAL.md §11.2; components/INTROGRESSION.md §6 stage 7). |
Stage 8. Stage → test → deploy
|
|
| What was done (10-03) |
built on the OSU staging copy /data/hop_data/osu_introgression_atlas/stage_viewer/introgression_atlas (port 8901). OA/scripts/o09_swap_into_served.sh did an md5-guarded copy into scratch_plots/introgression_atlas, backing up each file once as *.bak_2026-10-02_osu. Tests: for r in 5Mb 2Mb 1Mb 500kb; do bash OA/scripts/viewer_tests/run_tests.sh $r; done (p30 p15 p16 p11 p12 p07 k21 o21 o22), then p08_test_summary.py. All 36 PASS. |
| What to do now |
GENERIC: python3 hop_atlas/tools/atlas_layers.py stage --name <n> → apply the Stage 7 scripts with ATLAS_DIR=$STG (inferred) → atlas_layers.py serve $STG --port 8907 → atlas_layers.py test $STG --port 8907 → PI approval → atlas_layers.py deploy $STG → atlas_layers.py test scratch_plots/introgression_atlas --port 8900 (components/INTROGRESSION.md §2). If a cohort script rewrote index.html, run atlas_layers.py patch $STG. Use the stage tool, not o09: o09 is OSU-specific and references viewer_backup_pre_osu. |
| RNA lane test |
OA/scripts/viewer_tests/o22_wyerna_lane_test.py (W0 strings == call table with a must-fail shift; W1 groups and annotations; W2 ns == n_sites; W3 preset; W4 detail view). It is not in the generic suite sync/2026-10-07/lineage_refinement_v2/viewer/run_all_checks.sh that atlas_deploy.py test / atlas_layers.py test run. Run it by hand: RES=<res> python3 o22_wyerna_lane_test.py http://localhost:8907/introgression_atlas/ (inferred from run_tests.sh). |
| After deploy |
add a hop_atlas/CHANGELOG.md entry; register products (register_product.sh); update the catalog (Wye: products listed under osu_PRJNA1424574_introgression_atlas); write a sync handoff. |
3. Per-cohort differences
| Item |
Wye RNA (served) |
Apollo RNA (PRJEB63995) |
Comparator hop/Mimulus RNA |
| Read length / platform |
2×150, unstranded, poly-A |
2×100 HiSeq 2000 (ENA says NovaSeq) |
PE; mixed |
| Download |
aria2c + ENA md5 manifest |
dl_apollo_rnaseq.sh (wget, hybrid_atlas_urls.txt, 142 files = 71 runs) |
download_rnaseq.sh (prefetch / fasterq-dump) |
| Reference |
21110M (atlas), Apollo h1, MN586 |
not aligned (no record) |
drHumLupu1.1; Cascade for BRAKER3; E. guttata |
| Aligner |
CPU STAR 1-pass (21110M, Apollo h1); GPU pbrun 1-pass (MN586); GPU pbrun with chimeric (Lepton) |
n/a |
GPU pbrun 2-pass Basic, 2 GPUs |
| Duplicates |
marked; dropped in pileup and calls |
n/a |
marked |
| Parent identity |
RNA vs WGS gate (r05) + RNA F1 vs GBS F1 |
n/a (one plant) |
n/a |
| Atlas |
wyerna lane |
— |
— |
| Layers |
not in lineage COH (e03_viewer_export.py COH lacks wyerna); no sex lane, no phasing (ADDING_DATA.md) |
— |
— |
For a new cross RNA lane, edit:
- the manifest and sample names;
- the parent pair in r05, o16 and o15 meta();
- the WGS comparator call set (wye_bam → the cross's <cohort>_bam);
- the duplicate / mixture flags;
- the carrier annotations (Wye-specific; drop them);
- the preset name;
- the PANEL_ORDER groups;
- the pedigree RNA match basis (p01_pedigree_build.py, components/INTROGRESSION.md §3).
4. Missing, ambiguous or conflicting steps
- No generic pipeline. Every stage from 3a to 7 is a Wye/OSU-specific fork with hard-coded paths, sample names and parents (
ADDING_DATA.md gaps 1–3).
aria2_input.txt builder not recorded [manual].- No read QC / trimming step for RNA, and none was run.
- Alignment-route provenance conflict. Catalog
PRJNA694508_rnaseq_alignments says GPU pbrun for all arms. The logs show CPU STAR for 21110M/apollo_h1, and bam_21110M (the atlas input) is CPU STAR + samtools markdup. GPU vs CPU equivalence for the atlas was not tested on 21110M (mapping rates matched on MN586 only: 75.9 % vs 75.7 % unique).
- The technical doc is stale for wyerna.
annevo_eval/docs/INTROGRESSION_ATLAS_TECHNICAL.md lines 100, 139 and Appendix C2 still describe wyerna as "staged, not served, route V (VCF AD) from 21110M.rnaseq14.vcf.gz". The served lane is the o13 BAM-pileup build (OA/README.md §5, OA/PIPELINE.sh v3).
wyerna_samples.txt (/data/hop_data/osu_introgression_atlas/) creation is not scripted [manual].- EM refit policy when adding samples to an existing RNA cohort is undocumented.
- o15 cannot add to or update an existing lane (it asserts the cohort is absent and restarts from 10-02 backups). A re-export on today's served atlas needs a rewrite (inferred).
- o22 is not wired into
atlas_deploy.py / atlas_layers.py test.
- The RNA lane is absent from the lineage/subspecies layers, sex lane, phasing and dosage atlas. There is no record of any attempt.
r05_parent_identity_gate.sh and r00_driver.sh reference a worktree path (.claude/worktrees/chr5-inversion-survey/...). The main checkout has identical copies in sync/2026-10-01/wye_rnaseq_PRJNA694508/scripts/ (diff empty).- Apollo RNA-seq (71 runs) has never been aligned in any record found. The ANNEVO comparator BAMs, index and FASTQ are in
trash_rootdisk/ (PI: to be deleted).
- ASSAYS.md has no RNA-seq section. Candidate entries are all recorded in
W1/README.md:
- the sparse-SA refusal by pbrun;
--num-gpus 2 thrash;- the
zcat requirement;
- CRAM decoding by static StringTie (
r61 pipes through samtools);
- the variants-only trap applied to RNA.
- ENA provenance of the 8 ERR11670xxx runs is inconsistent between
RNASEQ_PROVENANCE.md (PRJEB63995) and catalog drhumlupu_rnaseq (DToL).
- Local data may be gone after today's PBSS archive. Restore
D/bam_21110M (or the FASTQ) from s3://general-purpose/skothenhill/archive/PRJNA694508/ before re-running o13. archive_ledger.json has no PRJNA694508 entry (grep, 2026-10-08).
Scripts worth generalising first:
- make_manifest.py (role/sex rule → manifest column);
- r01 + r02/r02c (one script, --route gpu|cpu, manifest-driven, any n);
- r05 gate (parent pair + WGS VCF as arguments);
- o13 (input glob + out dir as arguments; this is m01 WGS branch);
- o03 (OSU_BC → argument);
- o16 (pairs and duplicate pairs from a manifest);
- o15 (rows, groups, labels and preset from a cohort manifest, applied on top of the current stage instead of 10-02 backups);
- add o22 to run_all_checks.sh.