You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Copy file name to clipboardExpand all lines: docs/split-clip-audio-architecture-investigation.md
+15-15Lines changed: 15 additions & 15 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -10,23 +10,23 @@ This report investigates the current implementation without changing the existin
10
10
11
11
The native playback path is clip-ID-keyed decoded PCM, not a persistent decoder session keyed by source path.
12
12
13
-
The frontend builds one immutable native timeline snapshot from active audio clips, starts the native clock, clears the existing native clip list, and loads every snapshot clip concurrently ([nativeAudioTimeline.ts](/Users/AIEraDev/Documents/clypra-family/clypra/src/core/audio/nativeAudioTimeline.ts:59)). The Tauri load command opens/decodes one source path and installs the resulting clip into the clock mixer ([native_audio.rs](/Users/AIEraDev/Documents/clypra-family/clypra/src-tauri/src/commands/native_audio.rs:100)).
13
+
The frontend builds one immutable native timeline snapshot from active audio clips, starts the native clock, clears the existing native clip list, and loads every snapshot clip concurrently ([nativeAudioTimeline.ts](../src/core/audio/nativeAudioTimeline.ts:59)). The Tauri load command opens/decodes one source path and installs the resulting clip into the clock mixer ([native_audio.rs](../src-tauri/src/commands/native_audio.rs:100)).
14
14
15
-
Each decode opens a fresh FFmpeg input and creates a `DecodedAudioClip` containing the clip ID, timeline position, source start, duration, and PCM samples ([decoder.rs](/Users/AIEraDev/Documents/clypra-family/clypra/src-tauri/src/audio/decoder.rs:71), [mixer.rs](/Users/AIEraDev/Documents/clypra-family/clypra/src-tauri/src/audio/mixer.rs:75)). The native clock stores `NativePcmClip` values in a mixer vector and replaces/removes them by clip ID ([native_audio.rs](/Users/AIEraDev/Documents/clypra-family/clypra/src-tauri/src/native_audio.rs:116)). There is no native source-path decoder pool or decoder-position LRU.
15
+
Each decode opens a fresh FFmpeg input and creates a `DecodedAudioClip` containing the clip ID, timeline position, source start, duration, and PCM samples ([decoder.rs](../src-tauri/src/audio/decoder.rs:71), [mixer.rs](../src-tauri/src/audio/mixer.rs:75)). The native clock stores `NativePcmClip` values in a mixer vector and replaces/removes them by clip ID ([native_audio.rs](../src-tauri/src/native_audio.rs:116)). There is no native source-path decoder pool or decoder-position LRU.
16
16
17
17
### Boundary behavior
18
18
19
-
`source_start_ticks` is applied during each clip's initial decode. The corrected global seek and preroll trimming are therefore exercised for every clip whose `trimIn` is nonzero, including every right-hand split piece ([decoder.rs](/Users/AIEraDev/Documents/clypra-family/clypra/src-tauri/src/audio/decoder.rs:118)).
19
+
`source_start_ticks` is applied during each clip's initial decode. The corrected global seek and preroll trimming are therefore exercised for every clip whose `trimIn` is nonzero, including every right-hand split piece ([decoder.rs](../src-tauri/src/audio/decoder.rs:118)).
20
20
21
-
Continuous playback does not reposition a shared decoder at a boundary. The mixer calculates each clip's source position from `timeline_ticks - timeline_start_ticks` and samples the already-decoded PCM ([native_audio.rs](/Users/AIEraDev/Documents/clypra-family/clypra/src-tauri/src/native_audio.rs:230)). Explicit seeks only move the native timeline clock ([native_audio.rs](/Users/AIEraDev/Documents/clypra-family/clypra/src-tauri/src/native_audio.rs:628)); they do not re-seek or mutate an FFmpeg decoder.
21
+
Continuous playback does not reposition a shared decoder at a boundary. The mixer calculates each clip's source position from `timeline_ticks - timeline_start_ticks` and samples the already-decoded PCM ([native_audio.rs](../src-tauri/src/native_audio.rs:230)). Explicit seeks only move the native timeline clock ([native_audio.rs](../src-tauri/src/native_audio.rs:628)); they do not re-seek or mutate an FFmpeg decoder.
22
22
23
-
When a split is present, both halves can be installed simultaneously. They cannot interfere through a shared decoder cursor, but each half independently decodes its source range. The native mixer bounds resource usage at 64 clips and 512 MiB of PCM ([native_audio.rs](/Users/AIEraDev/Documents/clypra-family/clypra/src-tauri/src/native_audio.rs:10), [native_audio.rs](/Users/AIEraDev/Documents/clypra-family/clypra/src-tauri/src/native_audio.rs:151)). Native audio has no independent 2–3 second decoder lookahead; the separate video prewarm path is bounded at 1.5 seconds ([PreviewMediaPool.ts](/Users/AIEraDev/Documents/clypra-family/clypra/src/core/resources/PreviewMediaPool.ts:240)).
23
+
When a split is present, both halves can be installed simultaneously. They cannot interfere through a shared decoder cursor, but each half independently decodes its source range. The native mixer bounds resource usage at 64 clips and 512 MiB of PCM ([native_audio.rs](../src-tauri/src/native_audio.rs:10), [native_audio.rs](../src-tauri/src/native_audio.rs:151)). Native audio has no independent 2–3 second decoder lookahead; the separate video prewarm path is bounded at 1.5 seconds ([PreviewMediaPool.ts](../src/core/resources/PreviewMediaPool.ts:240)).
24
24
25
25
The source-path/session question is therefore resolved explicitly: native audio is keyed by clip ID after independent decode, not by a shared source-path decoder session. There is also no native audio lookahead buffer with a second decode path to validate. The reproduction suite covers the closest equivalent by installing a future clip before its timeline boundary and asserting that it remains silent until the boundary, then starts with the correct source segment.
26
26
27
27
### Browser fallback
28
28
29
-
The browser path is separate. Its decoded `AudioBuffer` is cached by media/source key, while active playback voices are keyed by clip ID ([AudioEngine.ts](/Users/AIEraDev/Documents/clypra-family/clypra/src/core/audio/AudioEngine.ts:142)). A shared full-source buffer is safe because each voice starts at `trimIn + timeIntoClip` ([AudioEngine.ts](/Users/AIEraDev/Documents/clypra-family/clypra/src/core/audio/AudioEngine.ts:158)). It does not reproduce the native shared-decoder-position bug, although it can share an incorrectly selected buffer if a clip's `audioPath` differs from the media asset while retaining the same `mediaId`.
29
+
The browser path is separate. Its decoded `AudioBuffer` is cached by media/source key, while active playback voices are keyed by clip ID ([AudioEngine.ts](../src/core/audio/AudioEngine.ts:142)). A shared full-source buffer is safe because each voice starts at `trimIn + timeIntoClip` ([AudioEngine.ts](../src/core/audio/AudioEngine.ts:158)). It does not reproduce the native shared-decoder-position bug, although it can share an incorrectly selected buffer if a clip's `audioPath` differs from the media asset while retaining the same `mediaId`.
30
30
31
31
### Classification
32
32
@@ -36,36 +36,36 @@ The native architecture is orthogonal to the original wrong-time-base bug. The f
36
36
37
37
### Call sites and range behavior
38
38
39
-
The `extract_waveform_data` Tauri command is invoked from one frontend hook ([useWaveformData.ts](/Users/AIEraDev/Documents/clypra-family/clypra/src/components/editor/timeline/useWaveformData.ts:90)); it is registered as a Tauri command ([lib.rs](/Users/AIEraDev/Documents/clypra-family/clypra/src-tauri/src/lib.rs:198)). Timeline video clips use `VolumeWaveform`, and audio clips use `TimelineWaveform`, with one waveform hook invocation per rendered clip ([Clip.tsx](/Users/AIEraDev/Documents/clypra-family/clypra/src/components/editor/timeline/Clip.tsx:815)).
39
+
The `extract_waveform_data` Tauri command is invoked from one frontend hook ([useWaveformData.ts](../src/components/editor/timeline/useWaveformData.ts:90)); it is registered as a Tauri command ([lib.rs](../src-tauri/src/lib.rs:198)). Timeline video clips use `VolumeWaveform`, and audio clips use `TimelineWaveform`, with one waveform hook invocation per rendered clip ([Clip.tsx](../src/components/editor/timeline/Clip.tsx:815)).
40
40
41
-
The native command passes FFmpeg `-ss` and `-t` from the requested `start_time` and `duration` ([media.rs](/Users/AIEraDev/Documents/clypra-family/clypra/src-tauri/src/commands/media.rs:499)). Therefore the native waveform decode is scoped to the clip's visible trimmed range rather than intentionally decoding the entire source file.
41
+
The native command passes FFmpeg `-ss` and `-t` from the requested `start_time` and `duration` ([media.rs](../src-tauri/src/commands/media.rs:499)). Therefore the native waveform decode is scoped to the clip's visible trimmed range rather than intentionally decoding the entire source file.
42
42
43
-
The browser fallback is different: `decodeAudioData` loads the complete source buffer, then JavaScript slices `startSample..endSample` for bucket computation ([useWaveformData.ts](/Users/AIEraDev/Documents/clypra-family/clypra/src/components/editor/timeline/useWaveformData.ts:118)). Thus the fallback performs full-file decode work even though the displayed waveform is trimmed.
43
+
The browser fallback is different: `decodeAudioData` loads the complete source buffer, then JavaScript slices `startSample..endSample` for bucket computation ([useWaveformData.ts](../src/components/editor/timeline/useWaveformData.ts:118)). Thus the fallback performs full-file decode work even though the displayed waveform is trimmed.
44
44
45
45
### Split and caching behavior
46
46
47
-
The split command creates two new clips, preserving the source path and assigning left/right trim ranges ([SplitClipCommand.ts](/Users/AIEraDev/Documents/clypra-family/clypra/src/core/history/commands/SplitClipCommand.ts:55)). The waveform cache key includes resolved path, source start, source duration, and bucket count ([useWaveformData.ts](/Users/AIEraDev/Documents/clypra-family/clypra/src/components/editor/timeline/useWaveformData.ts:79)). Consequently, a cold-cache split creates two waveform requests, one per resulting range. An N-way split creates N requests for N distinct ranges:
47
+
The split command creates two new clips, preserving the source path and assigning left/right trim ranges ([SplitClipCommand.ts](../src/core/history/commands/SplitClipCommand.ts:55)). The waveform cache key includes resolved path, source start, source duration, and bucket count ([useWaveformData.ts](../src/components/editor/timeline/useWaveformData.ts:79)). Consequently, a cold-cache split creates two waveform requests, one per resulting range. An N-way split creates N requests for N distinct ranges:
48
48
49
49
- Native/Tauri: N bounded-range FFmpeg decodes.
50
50
- Browser fallback: N full-source `decodeAudioData` operations followed by N range slices.
51
51
52
52
Exact duplicate requests can share the existing 50-entry LRU cache, but there is no coarse source-wide waveform cache analogous to the filmstrip L0 cache. This is an efficiency finding, not a playback-correctness defect.
53
53
54
-
The separate media-card waveform implementation also decodes through Web Audio, but it is not a call site of `extract_waveform_data` and is unrelated to split-clip timeline rendering ([MediaCardWaveform.tsx](/Users/AIEraDev/Documents/clypra-family/clypra/src/components/ui/cards/MediaCardWaveform.tsx:32)).
54
+
The separate media-card waveform implementation also decodes through Web Audio, but it is not a call site of `extract_waveform_data` and is unrelated to split-clip timeline rendering ([MediaCardWaveform.tsx](../src/components/ui/cards/MediaCardWaveform.tsx:32)).
55
55
56
56
## Part 3 — Detached audio isolation
57
57
58
-
`DetachAudioCommand` creates an audio-kind clip, sets `audioPath` to the selected source asset path, retains `detachedFromClipId`, and mutes the original video clip ([DetachAudioCommand.ts](/Users/AIEraDev/Documents/clypra-family/clypra/src/core/history/commands/DetachAudioCommand.ts:51)). Native timeline audio prioritizes `clip.audioPath` over the media asset path ([audioClips.ts](/Users/AIEraDev/Documents/clypra-family/clypra/src/core/timeline/audioClips.ts:82)). Waveform rendering also prioritizes the explicit path ([Clip.tsx](/Users/AIEraDev/Documents/clypra-family/clypra/src/components/editor/timeline/Clip.tsx:849)). For the built-in detach command, the explicit path currently equals the original video asset path by design.
58
+
`DetachAudioCommand` creates an audio-kind clip, sets `audioPath` to the selected source asset path, retains `detachedFromClipId`, and mutes the original video clip ([DetachAudioCommand.ts](../src/core/history/commands/DetachAudioCommand.ts:51)). Native timeline audio prioritizes `clip.audioPath` over the media asset path ([audioClips.ts](../src/core/timeline/audioClips.ts:82)). Waveform rendering also prioritizes the explicit path ([Clip.tsx](../src/components/editor/timeline/Clip.tsx:849)). For the built-in detach command, the explicit path currently equals the original video asset path by design.
59
59
60
-
Detached audio does not enter the visual compositor: the evaluator explicitly skips `clip.kind === "audio"` before creating media layers ([evaluator.ts](/Users/AIEraDev/Documents/clypra-family/clypra/src/core/evaluation/evaluator.ts:225)), and the preview media pool similarly excludes audio-kind clips from video bindings ([PreviewMediaPool.ts](/Users/AIEraDev/Documents/clypra-family/clypra/src/core/resources/PreviewMediaPool.ts:409)). Existing evaluator and preview-pool regression tests pass.
60
+
Detached audio does not enter the visual compositor: the evaluator explicitly skips `clip.kind === "audio"` before creating media layers ([evaluator.ts](../src/core/evaluation/evaluator.ts:225)), and the preview media pool similarly excludes audio-kind clips from video bindings ([PreviewMediaPool.ts](../src/core/resources/PreviewMediaPool.ts:409)). Existing evaluator and preview-pool regression tests pass.
61
61
62
62
### Latent divergence
63
63
64
-
The canonical evaluator's audio-layer construction requires an asset and resolves `sourcePath` from `asset.path`, ignoring `clip.audioPath` ([evaluator.ts](/Users/AIEraDev/Documents/clypra-family/clypra/src/core/evaluation/evaluator.ts:313)). This does not affect the current native preview path, which uses `getActiveAudioClips`, nor the current detach command, because both paths are equal today. It is a latent correctness gap for any audio-kind clip backed by a video asset whose explicit `audioPath` differs from that asset. No fix was applied during this investigation because the reproduced detach scenarios did not demonstrate wrong audible content in the active playback path.
64
+
The canonical evaluator's audio-layer construction requires an asset and resolves `sourcePath` from `asset.path`, ignoring `clip.audioPath` ([evaluator.ts](../src/core/evaluation/evaluator.ts:313)). This does not affect the current native preview path, which uses `getActiveAudioClips`, nor the current detach command, because both paths are equal today. It is a latent correctness gap for any audio-kind clip backed by a video asset whose explicit `audioPath` differs from that asset. No fix was applied during this investigation because the reproduced detach scenarios did not demonstrate wrong audible content in the active playback path.
65
65
66
66
## Part 4 — Reproduction matrix
67
67
68
-
The deterministic headless reproduction is implemented in [split_investigation_tests.rs](/Users/AIEraDev/Documents/clypra-family/clypra/src-tauri/src/audio/split_clip_investigation_tests.rs:1). It generates a temporary six-second A/V fixture with 440 Hz, 880 Hz, and 1760 Hz source regions, then validates source-frequency identity and amplitude.
68
+
The deterministic headless reproduction is implemented in [split_investigation_tests.rs](../src-tauri/src/audio/split_clip_investigation_tests.rs:1). It generates a temporary six-second A/V fixture with 440 Hz, 880 Hz, and 1760 Hz source regions, then validates source-frequency identity and amplitude.
Copy file name to clipboardExpand all lines: docs/text-effects-architecture.md
+2-2Lines changed: 2 additions & 2 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -75,7 +75,7 @@ The interpreter must be structurally incapable of executing anything beyond a wh
75
75
76
76
### 1.2 Offline, Caching & Export Policy
77
77
78
-
The desktop app and web studio must provide a reliable offline experience while strictly enforcing Clypra's [Architecture-First ADR](file:///Users/AIEraDev/Documents/clypra-family/clypra/docs/architecture-first-delivery-adr.md) (*"Silent fallback is not an acceptable production failure policy"*):
78
+
The desktop app and web studio must provide a reliable offline experience while strictly enforcing Clypra's [Architecture-First ADR](./architecture-first-delivery-adr.md) (*"Silent fallback is not an acceptable production failure policy"*):
0 commit comments