Return PNG spectrograms from FastAPI

[adrmac]

This issue has two levels of complexity. The first is straightforward, to produce a PNG spectrogram of a given stretch of audio using ffmpeg. The second is to carefully consider how we can display the spectrogram in a React interface.

Using the Google Pattern Radio site as a reference, we want to see buttery smooth rendering and zooming performance in the browser. Recommended approach from chatgpt research is as follows:

Spectrogram Tile Pyramid

Power Spectral Density is 1-dimensional: power vs frequency for a time interval.

If you compute a PSD for many tiny windows in time, you get a time series of PSDs → that’s basically a spectrogram (2D: time × frequency).

The catch: a long clip yields a huge 2D array. Shipping that whole float grid to the browser and resampling it on every zoom is slow and memory-heavy.

Existing approach:

• The Lambda+librosa tiler used in /orcasite works but feels slow/clunky for zooming. The fix is to separate compute from rendering and use a data tile pyramid:

The idea

• Server (one-time per clip): compute a high-res spectrogram once, build a multi-resolution pyramid (like map tiles), and save grayscale “data tiles” (uint8/uint16).
• Client (always): render tiles with WebGL and apply any colormap (viridis, magma, grayscale) instantly. No re-tiling to change colors/scales—just a new shader uniform.

Why this is faster

• Panning/zooming becomes a GPU paint problem (instant) using pre-baked tiles.
• You never recompute spectrograms on repeated views. “ffmpeg only once per clip” becomes true.
• Changing colormap or vmin/vmax happens client-side, not on Lambda.

Why a tile pyramid

Think Google Maps for spectrograms.

• You precompute the spectrogram once, then save it as many small tiles (e.g., 256×256 pixels) at multiple zoom levels (level 0 = very zoomed out; higher levels = more detail).
• In the browser you only download the few tiles needed for the current view/zoom, not the whole spectrogram.
• Result: instant pan/zoom, great caching, and no heavy resampling in the browser.

Tiles give you:

• Progressive loading (fast first paint),
• Only fetch what’s visible,
• Reuse cached tiles when panning,
• Compatible with CDNs.

What are uint8 / uint16 tiles?
They’re just compact numbers instead of big floats:

• Start from per-bin power → dB.
• Map dB to bytes: e.g., [-100 dB … 0 dB] → 0…255 (uint8, 1 byte). (Use uint16 if you want more precision.)
• Result: each pixel = one time–frequency cell’s dB, quantized to a byte.

Not PSD “averages” per se
Native (max zoom) tiles: each byte is a single STFT time–freq bin (no averaging).

Lower zoom levels: each pixel represents an average of a small block of native bins (e.g., 2×2 or 4×4) — average in power, then convert to dB, then quantize. That’s local downsampling for zoomed-out views, not a whole-clip PSD.

Why bother with bytes?

Size & speed:

• Example (10 min @ 48 kHz, n_fft=2048, hop=512): ~1,025 freq bins × 56,250 frames ≈ 57.6M cells.
• float32: ~230 MB
• uint8: ~57 MB (¼ the size) — and with tiling + gzip + only fetching what’s visible, you’re typically loading MBs, not hundreds of MBs.

Bytes are perfect for rendering; you keep full-precision PSD/metrics separately for science.

Why not “PSD Parquet → canvas”?

• Parquet is great for analytics, not interactive viewing in the browser.
• To draw a spectrogram from Parquet, you’d still need to load massive chunks and downsample on the fly each zoom/pan — slow and memory-hungry compared to just pulling a few precomputed tiles.
• WebGL viewer (what it is, why it’s smooth)
• WebGL lets the GPU paint the grayscale byte tiles at 60 fps.
• Recoloring or changing scale is just a shader uniform change — instant, no re-tiling.

What is a “WebGL viewer”?

• WebGL = the browser’s GPU API. A “WebGL viewer” is a small JS canvas that asks the GPU to render the tiles.
• Why it matters: the GPU draws/reshades tiles at 60 fps and recolors by changing a few uniforms (no re-tiling).
• You can build this with raw WebGL2, or helpers like regl, deck.gl, or even WebGPU. (Canvas 2D works for small cases, but WebGL scales better.)

Summary

• A spectrogram tile pyramid = precomputed 2D spectrogram saved as small byte tiles across zoom levels.
• uint8/uint16 tiles store quantized dB per time–freq cell (native) or small averages (lower zooms).
• This makes pan/zoom fast and lets you recolor instantly, while you compute/keep PSD + metrics separately for scientific correctness.