diff --git a/07 - Libretro API Testing and Validation.md b/07 - Libretro API Testing and Validation.md new file mode 100644 index 000000000..d251aeb9a --- /dev/null +++ b/07 - Libretro API Testing and Validation.md @@ -0,0 +1,167 @@ +# Milestone 07 - Libretro API Testing and Validation + +**Status:** ✅ Complete +**Date:** October 30, 2025 + +## Objective + +Validate that the parallel-n64 libretro core compiled to WebAssembly correctly exposes and implements the essential libretro API functions in a browser environment. + +## Initial Issues + +When first running `test_libretro.html`, encountered multiple failures: + +1. **Export Issues:** + - `Module._malloc is not a function` - malloc/free not exported + - `Module.addFunction is not a function` - addFunction not exported + - `Module.UTF8ToString` not available + - `Module.HEAPU8` / `Module.HEAPU32` not available + +2. **API Usage Issues:** + - `retro_init()` failing with "null function or function signature mismatch" + - Root cause: `environ_cb` was NULL because `retro_set_environment()` was never called + +3. **Browser Caching:** + - Updated WASM builds not loading due to aggressive browser caching + +## Solutions Implemented + +### 1. Updated Makefile Exports (lines 502-503) + +**Added to EXPORTED_FUNCTIONS:** +```makefile +"_malloc","_free" +``` + +**Updated EXPORTED_RUNTIME_METHODS:** +```makefile +'["ccall","cwrap","UTF8ToString","addFunction","HEAPU8","HEAPU32"]' +``` + +**Added linker flag:** +```makefile +-s ALLOW_TABLE_GROWTH=1 +``` +This enables dynamic function table growth required by `addFunction()`. + +### 2. Fixed Test Sequence in test_libretro.html + +**Added Test 3: retro_set_environment()** +```javascript +test('retro_set_environment()', () => { + // Create a dummy environment callback + const envCallback = Module.addFunction((cmd, data) => { + return 0; // false - minimal implementation + }, 'iii'); // return int, params: int, pointer + + Module._retro_set_environment(envCallback); + return 'Environment callback set successfully'; +}); +``` + +This ensures `environ_cb` is set before `retro_init()` tries to use it (libretro/libretro.c:1011-1024). + +**Key insight:** The libretro API requires callbacks to be set in a specific order: +1. `retro_set_environment()` - MUST be called first +2. `retro_init()` - Uses environment callback during initialization +3. Other setup functions +4. `retro_load_game()` - Load actual content + +### 3. Fixed Memory Access + +Changed from `Module.HEAP8` (not exported) to `Module.HEAPU8` (now exported): +```javascript +for (let i = 0; i < structSize; i++) { + Module.HEAPU8[ptr + i] = 0; // Was: Module.HEAP8 +} +``` + +### 4. UI Polish + +Made "Failed: 0" display in green to indicate success: +```javascript +Failed: ${testsFailed} +``` + +## Test Results + +All 6 tests now pass: + +1. ✅ **retro_api_version()** - Returns API version 1 +2. ✅ **retro_get_system_info()** - Retrieves library name and version using malloc +3. ✅ **retro_set_environment()** - Sets up environment callback using addFunction +4. ✅ **retro_init()** - Initializes core successfully +5. ✅ **retro_deinit()** - Cleans up core resources +6. ✅ **retro_init() again** - Re-initializes for further testing + +## File Sizes + +- **parallel_n64_libretro.js:** 187 KB +- **parallel_n64_libretro.wasm:** 1.4 MB +- **Total:** ~1.6 MB + +## Key Learnings + +### Libretro API Lifecycle +The libretro specification requires this initialization order: +``` +retro_set_environment() → Sets callbacks +retro_init() → Uses environ_cb internally +retro_load_game() → Loads content +retro_run() → Main loop +retro_unload_game() → Cleanup +retro_deinit() → Final cleanup +``` + +### Emscripten Function Tables +- `addFunction()` requires `-s ALLOW_TABLE_GROWTH=1` +- Function signature must be specified: `'iii'` = returns int, takes (int, int) +- Wrapped functions can be called from C code as function pointers + +### Memory Views in WASM +- Memory views must be explicitly exported via `EXPORTED_RUNTIME_METHODS` +- `HEAPU8` = unsigned byte access (most common) +- `HEAPU32` = unsigned 32-bit access (for pointer arrays) +- Memory views are automatically updated when WASM memory grows + +## Browser Testing Notes + +- **Cache Management:** Disable caching during development or use cache-busting query params +- **Console Logging:** All stdout/stderr from WASM appears in browser console +- **Memory Inspection:** Can examine WASM memory directly via Module.HEAPU8/32 + +## Code Locations + +- **Makefile:** Emscripten configuration (lines 482-511) +- **test_libretro.html:** Browser test harness +- **libretro/libretro.c:** Core API implementation + - `retro_init()` at line 1003 + - `retro_set_environment()` usage at line 1011 + +## Next Steps + +Potential future milestones: + +1. **ROM Loading Test** - Test `retro_load_game()` with actual N64 ROM +2. **Graphics Context** - Set up WebGL2 context for rendering +3. **Input Handling** - Implement gamepad/keyboard input callbacks +4. **Audio Output** - Set up Web Audio API for sound +5. **Frame Rendering** - Test actual emulation loop (`retro_run()`) +6. **Save States** - Test serialization/deserialization +7. **Performance Profiling** - Measure frame rates and optimize + +## Validation Checklist + +- [x] Core loads in browser without errors +- [x] All libretro API functions are callable from JavaScript +- [x] Memory allocation/deallocation works +- [x] String data can be read from WASM memory +- [x] JavaScript callbacks can be passed to C code +- [x] Core initialization/cleanup works correctly +- [x] Multiple init/deinit cycles succeed + +## Conclusion + +The parallel-n64 libretro core successfully runs in WebAssembly and exposes a working JavaScript API. All fundamental libretro API functions are operational and can be called from browser JavaScript. The core is ready for the next phase: actual ROM loading and emulation testing. + +**This milestone demonstrates that the core C/C++ emulation code compiles to WASM and maintains API compatibility with the libretro specification.** diff --git a/10 - Input + Audio.md b/10 - Input + Audio.md new file mode 100644 index 000000000..ac825dcee --- /dev/null +++ b/10 - Input + Audio.md @@ -0,0 +1,648 @@ +# Milestone 10 - Input + Audio + +**Status:** 🔄 Pending +**Date:** October 30, 2025 + +## Objective + +Implement real input handling (keyboard and gamepad) and audio output (Web Audio API) to create a fully interactive N64 emulator running in the browser. Replace stub implementations from Milestone 09 with functional callbacks and establish a continuous emulation loop. + +## Prerequisites + +- ✅ Milestone 09 complete - First frame rendered successfully +- ✅ WebGL2 rendering working +- ✅ ROM loaded and running +- ✅ Modern browser with Web Audio API and Gamepad API support + +## Implementation Steps + +### Part A: Input Implementation + +#### 1. N64 Controller Button Mapping + +N64 controller layout (from libretro API): +```javascript +const N64_BUTTONS = { + RETRO_DEVICE_ID_JOYPAD_A: 8, // A button + RETRO_DEVICE_ID_JOYPAD_B: 0, // B button + RETRO_DEVICE_ID_JOYPAD_START: 3, // Start + RETRO_DEVICE_ID_JOYPAD_UP: 4, // D-Pad Up + RETRO_DEVICE_ID_JOYPAD_DOWN: 5, // D-Pad Down + RETRO_DEVICE_ID_JOYPAD_LEFT: 6, // D-Pad Left + RETRO_DEVICE_ID_JOYPAD_RIGHT: 7, // D-Pad Right + RETRO_DEVICE_ID_JOYPAD_L: 10, // L trigger + RETRO_DEVICE_ID_JOYPAD_R: 11, // R trigger + RETRO_DEVICE_ID_JOYPAD_L2: 12, // Z trigger (mapped to L2) + RETRO_DEVICE_ID_JOYPAD_X: 1, // C-Up (mapped to X) + RETRO_DEVICE_ID_JOYPAD_Y: 14, // C-Down (mapped to Y) + RETRO_DEVICE_ID_JOYPAD_L3: 15, // C-Left (mapped to L3) + RETRO_DEVICE_ID_JOYPAD_R3: 16, // C-Right (mapped to R3) + // Analog stick via RETRO_DEVICE_ANALOG +}; +``` + +#### 2. Keyboard Input State + +```javascript +const inputState = { + buttons: new Set(), // Pressed buttons + analogX: 0, // -32768 to 32767 + analogY: 0 // -32768 to 32767 +}; + +// Keyboard mappings +const keyMap = { + 'KeyX': 8, // A + 'KeyZ': 0, // B + 'Enter': 3, // Start + 'ArrowUp': 4, // D-Up + 'ArrowDown': 5, // D-Down + 'ArrowLeft': 6, // D-Left + 'ArrowRight': 7,// D-Right + 'KeyA': 10, // L + 'KeyS': 11, // R + 'KeyQ': 12, // Z + 'KeyI': 1, // C-Up + 'KeyK': 14, // C-Down + 'KeyJ': 15, // C-Left + 'KeyL': 16 // C-Right +}; + +window.addEventListener('keydown', (e) => { + if (keyMap[e.code] !== undefined) { + inputState.buttons.add(keyMap[e.code]); + e.preventDefault(); + } + // Arrow keys for analog stick + if (e.code === 'ArrowUp') inputState.analogY = -32767; + if (e.code === 'ArrowDown') inputState.analogY = 32767; + if (e.code === 'ArrowLeft') inputState.analogX = -32767; + if (e.code === 'ArrowRight') inputState.analogX = 32767; +}); + +window.addEventListener('keyup', (e) => { + if (keyMap[e.code] !== undefined) { + inputState.buttons.delete(keyMap[e.code]); + } + if (e.code === 'ArrowUp' || e.code === 'ArrowDown') inputState.analogY = 0; + if (e.code === 'ArrowLeft' || e.code === 'ArrowRight') inputState.analogX = 0; +}); +``` + +#### 3. Gamepad API Support + +```javascript +function updateGamepad() { + const gamepads = navigator.getGamepads(); + const gamepad = gamepads[0]; // Use first connected gamepad + + if (!gamepad) return; + + // Map gamepad buttons to N64 buttons + inputState.buttons.clear(); + if (gamepad.buttons[0]?.pressed) inputState.buttons.add(8); // A + if (gamepad.buttons[1]?.pressed) inputState.buttons.add(0); // B + if (gamepad.buttons[9]?.pressed) inputState.buttons.add(3); // Start + // ... map remaining buttons + + // Analog stick (axis 0 = X, axis 1 = Y) + inputState.analogX = Math.round(gamepad.axes[0] * 32767); + inputState.analogY = Math.round(gamepad.axes[1] * 32767); +} +``` + +#### 4. Input Callbacks Implementation + +```javascript +const inputPollCallback = Module.addFunction(() => { + // Update input state from keyboard/gamepad + updateGamepad(); +}, 'v'); + +const inputStateCallback = Module.addFunction((port, device, index, id) => { + // port: controller port (0-3) + // device: RETRO_DEVICE_JOYPAD or RETRO_DEVICE_ANALOG + // index: for analog (0=left stick, 1=right stick) + // id: button ID or axis ID + + if (port !== 0) return 0; // Only port 0 supported + + if (device === 1) { // RETRO_DEVICE_JOYPAD + return inputState.buttons.has(id) ? 1 : 0; + } else if (device === 2) { // RETRO_DEVICE_ANALOG + if (index === 0) { // Left stick + if (id === 0) return inputState.analogX; // X axis + if (id === 1) return inputState.analogY; // Y axis + } + } + + return 0; +}, 'iiiii'); +``` + +### Part B: Audio Implementation + +#### 1. Web Audio Context Setup + +```javascript +const audioContext = new (window.AudioContext || window.webkitAudioContext)({ + sampleRate: 44100, // Match N64 audio output + latencyHint: 'interactive' +}); + +// Create audio worklet or script processor for streaming +const audioBufferSize = 4096; +const audioQueue = []; +let audioStartTime = 0; +``` + +#### 2. Audio Sample Callback + +The core can call audio callbacks in two ways: +- `audio_sample(left, right)` - single stereo sample +- `audio_sample_batch(data, frames)` - batch of samples + +```javascript +const audioSampleCallback = Module.addFunction((left, right) => { + // Convert int16 samples to float32 [-1.0, 1.0] + const leftFloat = (left << 16 >> 16) / 32768.0; // Sign extend + const rightFloat = (right << 16 >> 16) / 32768.0; + + audioQueue.push(leftFloat, rightFloat); + + // Flush to Web Audio when buffer is full + if (audioQueue.length >= audioBufferSize * 2) { + flushAudioBuffer(); + } +}, 'vii'); + +const audioSampleBatchCallback = Module.addFunction((data, frames) => { + // data = pointer to int16 stereo samples + // frames = number of stereo sample pairs + + for (let i = 0; i < frames; i++) { + const left = Module.HEAP16[data/2 + i*2]; + const right = Module.HEAP16[data/2 + i*2 + 1]; + + audioQueue.push(left / 32768.0, right / 32768.0); + } + + if (audioQueue.length >= audioBufferSize * 2) { + flushAudioBuffer(); + } + + return frames; +}, 'iii'); +``` + +#### 3. Audio Buffer Management + +```javascript +function flushAudioBuffer() { + if (audioQueue.length === 0) return; + + // Create audio buffer + const frames = audioQueue.length / 2; + const buffer = audioContext.createBuffer(2, frames, audioContext.sampleRate); + + // Fill left and right channels + const leftChannel = buffer.getChannelData(0); + const rightChannel = buffer.getChannelData(1); + + for (let i = 0; i < frames; i++) { + leftChannel[i] = audioQueue[i * 2]; + rightChannel[i] = audioQueue[i * 2 + 1]; + } + + // Schedule playback + const source = audioContext.createBufferSource(); + source.buffer = buffer; + source.connect(audioContext.destination); + + const playTime = Math.max(audioContext.currentTime, audioStartTime); + source.start(playTime); + audioStartTime = playTime + buffer.duration; + + // Clear queue + audioQueue.length = 0; +} +``` + +#### 4. Register Audio Callbacks + +```javascript +Module._retro_set_audio_sample(audioSampleCallback); +Module._retro_set_audio_sample_batch(audioSampleBatchCallback); +``` + +**Note:** Need to export `retro_set_audio_sample_batch` in Makefile if not already exported. + +### Part C: Continuous Emulation Loop + +#### 1. RequestAnimationFrame Loop + +```javascript +let isRunning = false; +let frameCount = 0; +let lastTime = performance.now(); + +function emulationLoop() { + if (!isRunning) return; + + // Run one frame of emulation + Module._retro_run(); + + frameCount++; + + // FPS counter + const now = performance.now(); + if (now - lastTime >= 1000) { + console.log(`FPS: ${frameCount}`); + frameCount = 0; + lastTime = now; + } + + // Schedule next frame (target 60 FPS) + requestAnimationFrame(emulationLoop); +} + +// Start emulation +isRunning = true; +emulationLoop(); +``` + +#### 2. Pause/Resume Controls + +```html + + +``` + +```javascript +document.getElementById('pauseBtn').addEventListener('click', () => { + isRunning = false; +}); + +document.getElementById('resumeBtn').addEventListener('click', () => { + if (!isRunning) { + isRunning = true; + lastTime = performance.now(); + emulationLoop(); + } +}); +``` + +## Expected Outcomes + +### Success Criteria + +- ✅ Keyboard input controls N64 emulator +- ✅ Gamepad input recognized and mapped correctly +- ✅ Audio plays without crackling or stuttering +- ✅ Emulation runs at consistent ~60 FPS +- ✅ Game responds to controller input (menu navigation, gameplay) +- ✅ Audio synchronized with video +- ✅ Pause/resume functionality works + +### Expected Console Output + +``` +Audio context initialized: 44100 Hz +Input handlers registered +Starting emulation loop... +FPS: 60 +FPS: 59 +FPS: 60 +Audio buffer flushed: 2048 samples +Gamepad connected: Xbox Controller +FPS: 60 +``` + +### Expected Behavior + +**WaveRace.n64 should:** +1. Display Nintendo 64 boot logo with sound +2. Show game title screen with music +3. Navigate menus with D-pad/analog stick +4. Select options with A button +5. Start gameplay with interactive controls +6. Play game audio and sound effects +7. Render at smooth ~60 FPS + +### Potential Issues + +1. **Audio crackling:** + - Buffer underruns (queue too small) + - Wrong sample rate (N64 uses ~32000-48000 Hz) + - Need larger `audioBufferSize` + +2. **Input lag:** + - `inputPollCallback` not called frequently enough + - Gamepad API polling delay + +3. **Performance drops:** + - WASM execution too slow + - Graphics plugin overhead + - Disable debugging output for better performance + +4. **Audio/video desync:** + - Core expects audio pacing for timing + - May need frame skipping logic + +5. **Button mapping confusion:** + - N64 has unique button layout (C-buttons, Z trigger) + - Need clear on-screen guide + +## File Modifications + +**test_libretro.html:** +- Add keyboard event listeners +- Add gamepad polling +- Add Web Audio setup +- Add audio sample callbacks +- Add emulation loop +- Add pause/resume controls +- Add on-screen control guide + +**Makefile:** +- Verify `retro_set_audio_sample_batch` is exported +- May need `-s EXPORTED_FUNCTIONS` update + +## Validation Tests + +1. ✅ **Keyboard Input** - Press keys, verify button state changes +2. ✅ **Gamepad Detection** - Connect gamepad, verify recognized +3. ✅ **Analog Stick** - Move stick, verify in-game response +4. ✅ **Audio Playback** - Hear boot sound and music +5. ✅ **60 FPS Sustained** - Monitor FPS counter over 30 seconds +6. ✅ **Menu Navigation** - Use D-pad to navigate game menus +7. ✅ **Gameplay** - Start race in WaveRace, control vehicle +8. ✅ **Pause/Resume** - Verify emulation pauses correctly + +## Performance Expectations + +- **Target FPS:** 60 (may drop to 50-55 on complex scenes) +- **Audio latency:** <100ms +- **Input latency:** <50ms (2-3 frames) +- **CPU usage:** 30-80% (single core) + +## Debugging Tips + +1. **Test input without emulation:** + ```javascript + console.log('Button state:', Array.from(inputState.buttons)); + ``` + +2. **Monitor audio queue:** + ```javascript + console.log('Audio queue size:', audioQueue.length); + ``` + +3. **Measure frame time:** + ```javascript + const start = performance.now(); + Module._retro_run(); + console.log('Frame time:', performance.now() - start, 'ms'); + ``` + +4. **Check for WASM errors:** + - Look for exceptions in console + - Check for memory access violations + +## Code Locations + +- **libretro/libretro.c:** + - `retro_set_input_poll()` (~line 950) + - `retro_set_input_state()` (~line 955) + - `retro_set_audio_sample()` (~line 960) +- **mupen64plus-core/src/plugin/plugin.h:** + - Input plugin interface +- **mupen64plus-core/src/pi/pi_controller.c:** + - N64 controller emulation + +## Milestone 11: Audio-Sync Timing Implementation + +**Status:** ✅ Complete +**Date:** October 31, 2025 + +### Objective + +Replace requestAnimationFrame-based timing with audio-driven frame generation for more accurate emulation speed and better audio/video synchronization. This approach eliminates timing drift by locking emulation to the audio hardware clock. + +### Background + +The N64Wasm project demonstrated that using the Web Audio callback to drive frame generation provides superior timing accuracy compared to RAF. The audio hardware clock runs at a precise sample rate (44100Hz), while RAF can vary between monitors (59.94Hz, 60Hz, 75Hz, 144Hz). + +### Implementation + +#### 1. Ring Buffer System + +Replaced the old "queue and flush" audio system with a ring buffer: + +```javascript +const AUDIO_RING_BUFFER_SIZE = 64000; // samples per channel +const AUDIO_CALLBACK_BUFFER_SIZE = 1024; // samples per audio callback +let audioRingBuffer = new Float32Array(AUDIO_RING_BUFFER_SIZE * 2); // stereo +let audioWritePosition = 0; +let audioReadPosition = 0; + +function writeToRingBuffer(leftSample, rightSample) { + const writeIdx = audioWritePosition * 2; // stereo + audioRingBuffer[writeIdx] = leftSample; + audioRingBuffer[writeIdx + 1] = rightSample; + audioWritePosition = (audioWritePosition + 1) % AUDIO_RING_BUFFER_SIZE; +} + +function hasEnoughSamples() { + const available = (audioWritePosition - audioReadPosition + AUDIO_RING_BUFFER_SIZE) % AUDIO_RING_BUFFER_SIZE; + return available >= AUDIO_CALLBACK_BUFFER_SIZE; +} +``` + +#### 2. ScriptProcessor Audio Callback + +Created an audio callback that drives frame generation: + +```javascript +scriptProcessor = audioContext.createScriptProcessor( + AUDIO_CALLBACK_BUFFER_SIZE, + 2, // 2 input channels + 2 // 2 output channels (stereo) +); + +function audioProcessCallback(event) { + const outputBuffer = event.outputBuffer; + const outputL = outputBuffer.getChannelData(0); + const outputR = outputBuffer.getChannelData(1); + + // If audio-sync is enabled, generate frames as needed + if (audioSyncEnabled && isContinuous) { + let attempts = 0; + while (!hasEnoughSamples() && attempts < 3) { + Module._retro_run(); + frameCount++; + fpsFrameCount++; + attempts++; + } + } + + // Copy samples from ring buffer to output + for (let i = 0; i < AUDIO_CALLBACK_BUFFER_SIZE; i++) { + const available = (audioWritePosition - audioReadPosition + AUDIO_RING_BUFFER_SIZE) % AUDIO_RING_BUFFER_SIZE; + + if (available > 0) { + const readIdx = audioReadPosition * 2; + outputL[i] = audioRingBuffer[readIdx]; + outputR[i] = audioRingBuffer[readIdx + 1]; + audioReadPosition = (audioReadPosition + 1) % AUDIO_RING_BUFFER_SIZE; + } else { + // Buffer underrun - output silence + outputL[i] = 0; + outputR[i] = 0; + } + } +} +``` + +#### 3. Dual Timing Modes + +Implemented both timing strategies: + +```javascript +function toggleContinuous() { + isContinuous = !isContinuous; + + if (isContinuous) { + if (audioSyncEnabled && audioInitialized) { + log('Starting continuous emulation with AUDIO-SYNC timing...'); + rafForDisplayOnly(); // RAF only updates FPS display + } else { + log('Starting continuous emulation with RAF timing (classic mode)...'); + continuousLoopRAF(); // Traditional RAF-driven loop + } + } else { + // Stop emulation + } +} +``` + +#### 4. User Interface Control + +Added a checkbox to toggle audio-sync mode: + +```html +
Testing parallel-n64 libretro core compiled to WebAssembly.
+ ++Arrow Keys = D-Pad / Analog Stick +X = A Button +Z = B Button +Enter = Start +A = L Trigger +S = R Trigger +Q = Z Trigger +I/K/J/L = C-Buttons (Up/Down/Left/Right)+
+D-Pad/Left Stick = D-Pad / Analog +Button 0 (A) = A Button +Button 1 (B) = B Button +Button 9 (Start) = Start +L1/L2 = L/Z Triggers +R1 = R Trigger +Right Stick = C-Buttons+