Skip to content

Conversation

@kixelated
Copy link
Collaborator

@kixelated kixelated commented Aug 27, 2025

feels bad man

Summary by CodeRabbit

  • New Features

    • Added configurable audio latency.
    • Exposed captions and speaking controls in the audio viewer.
  • Refactor

    • Switched audio/video processing to direct worker URLs for capture, render, and detection.
    • Removed legacy service-worker-based worklet loader.
  • Chores

    • Updated monorepo dependencies to workspace references for unified resolution.
    • Removed Prettier from dev dependencies.

@kixelated kixelated enabled auto-merge (squash) August 27, 2025 22:26
@coderabbitai
Copy link
Contributor

coderabbitai bot commented Aug 27, 2025

Walkthrough

Replaces dynamic service-worker-based worklet/worker loading with Vite worker URL imports across audio and video paths, removes the loadAudioWorklet hack, updates package manifests, and expands watch/audio API to include latency, captions, and speaking while initializing the render worklet via AudioContext.addModule.

Changes

Cohort / File(s) Summary
Package manifest updates
js/hang/package.json, js/package.json
Switches @kixelated/moq and @kixelated/signals from link:… to workspace:^; reorders dependencies; removes devDependency prettier in js/package.json.
Publish audio worklet import refactor
js/hang/src/publish/audio/captions.ts, js/hang/src/publish/audio/index.ts, js/hang/src/publish/audio/speaking.ts
Replaces dynamic loadAudioWorklet approach with Vite worker URL imports (./capture-worklet?worker&url) and uses audioWorklet.addModule(...). No API changes.
Watch audio API and init changes
js/hang/src/watch/audio/index.ts
Adds AudioProps (latency, captions, speaking). Exposes captions, speaking, and readonly latency. Loads render worklet via Vite URL import and passes latency in init message. Removes loadAudioWorklet usage.
Video detection worker import refactor
js/hang/src/publish/video/detection.ts
Replaces dynamic worker URL creation with Vite worker URL import (./detection-worker?worker&url) and instantiation. Flow unchanged.
Utility hack removal
js/hang/src/util/hacks.ts
Removes loadAudioWorklet and related async-mutex hack; module now exports only isChrome and isFirefox.

Sequence Diagram(s)

sequenceDiagram
  autonumber
  actor App as App
  participant Audio as Audio (watch)
  participant AC as AudioContext
  participant RW as RenderWorklet (module)
  participant AWN as AudioWorkletNode

  App->>Audio: new Audio(stream, props{ latency, captions, speaking })
  Audio->>AC: create AudioContext
  Note over Audio,AC: Vite-only import of render worklet URL
  Audio->>AC: audioWorklet.addModule(RenderWorklet URL)
  AC-->>Audio: module loaded
  Audio->>AC: new AudioWorkletNode("render")
  Audio->>AWN: postMessage({ type:"init", latency })
  par Subsystems
    Audio->>Audio: this.captions = new Captions(...)
    Audio->>Audio: this.speaking = new Speaking(...)
  end
Loading

Estimated code review effort

🎯 3 (Moderate) | ⏱️ ~25 minutes

Possibly related PRs

Poem

A hare taps code with nimble might,
Worklets hop in Vite’s soft light.
No hacks to herd, no mutex maze,
Latency set—syncing plays.
Captions whisper, speech takes wing—
Carrots compiled, ship the thing! 🥕✨

Tip

🔌 Remote MCP (Model Context Protocol) integration is now available!

Pro plan users can now connect to remote MCP servers from the Integrations page. Connect with popular remote MCPs such as Notion and Linear to add more context to your reviews and chats.

✨ Finishing Touches
  • 📝 Generate Docstrings
🧪 Generate unit tests
  • Create PR with unit tests
  • Post copyable unit tests in a comment
  • Commit unit tests in branch revert-webpack

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share
🪧 Tips

Chat

There are 3 ways to chat with CodeRabbit:

  • Review comments: Directly reply to a review comment made by CodeRabbit. Example:
    • I pushed a fix in commit <commit_id>, please review it.
    • Open a follow-up GitHub issue for this discussion.
  • Files and specific lines of code (under the "Files changed" tab): Tag @coderabbitai in a new review comment at the desired location with your query.
  • PR comments: Tag @coderabbitai in a new PR comment to ask questions about the PR branch. For the best results, please provide a very specific query, as very limited context is provided in this mode. Examples:
    • @coderabbitai gather interesting stats about this repository and render them as a table. Additionally, render a pie chart showing the language distribution in the codebase.
    • @coderabbitai read the files in the src/scheduler package and generate a class diagram using mermaid and a README in the markdown format.

Support

Need help? Create a ticket on our support page for assistance with any issues or questions.

CodeRabbit Commands (Invoked using PR/Issue comments)

Type @coderabbitai help to get the list of available commands.

Other keywords and placeholders

  • Add @coderabbitai ignore anywhere in the PR description to prevent this PR from being reviewed.
  • Add @coderabbitai summary to generate the high-level summary at a specific location in the PR description.
  • Add @coderabbitai anywhere in the PR title to generate the title automatically.

CodeRabbit Configuration File (.coderabbit.yaml)

  • You can programmatically configure CodeRabbit by adding a .coderabbit.yaml file to the root of your repository.
  • Please see the configuration documentation for more information.
  • If your editor has YAML language server enabled, you can add the path at the top of this file to enable auto-completion and validation: # yaml-language-server: $schema=https://coderabbit.ai/integrations/schema.v2.json

Status, Documentation and Community

  • Visit our Status Page to check the current availability of CodeRabbit.
  • Visit our Documentation for detailed information on how to use CodeRabbit.
  • Join our Discord Community to get help, request features, and share feedback.
  • Follow us on X/Twitter for updates and announcements.

@claude
Copy link
Contributor

claude bot commented Aug 27, 2025

Claude finished @kixelated's task —— View job


Code Review - Revert Webpack Support

Todo List:

  • Review changed files for bugs and security issues
  • Check package.json changes for consistency
  • Verify revert is clean and complete
  • Provide final assessment

Review Summary

I've reviewed this revert PR that removes webpack support functionality. The changes look clean and well-executed. Here are my findings:

No Issues Found

Changes Analysis:

  • Reverts webpack compatibility layer in js/hang/src/util/hacks.ts (removed 24 lines of webpack-specific code)
  • Restores Vite-exclusive worker imports using ?worker&url syntax across audio/video modules
  • Updates package.json dependencies from link: to workspace:^ references
  • Removes webpack-related dependency (prettier)
  • All import statements are properly updated and consistent

Security & Code Quality:

  • No security vulnerabilities introduced
  • No broken imports or orphaned references
  • Package dependency changes are clean and consistent
  • Worker loading mechanism reverted to simpler, more reliable approach

The revert appears complete and doesn't leave any webpack-specific code artifacts behind. The change from the complex service worker workaround back to direct Vite worker imports actually improves code maintainability.

👍 This revert is safe to merge.

Copy link
Contributor

@coderabbitai coderabbitai bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 0

🧹 Nitpick comments (12)
js/hang/src/publish/video/detection.ts (2)

55-56: Minor: name the worker and add an error handler for easier debugging.

Apply within these lines:

-const worker = new Worker(WorkerUrl, { type: "module" });
+const worker = new Worker(WorkerUrl, { type: "module", name: "detection" });

Add right after creation (outside this hunk):

worker.addEventListener("error", (e) =>
  console.error("detection worker error:", e.error ?? e.message)
);

71-84: Avoid unhandled rejections on teardown by racing detect with cancellation.

Current flow schedules process() after ready(), but if the effect cleans up while api.detect() awaits, terminating the worker can throw. Wrap each detect in a cancel race and guard rescheduling:

effect.spawn(async (cancel) => {
  const ready = await Promise.race([api.ready(), cancel]);
  if (!ready) return;

  const process = async (): Promise<void> => {
    const frame = this.video.frame.peek();
    if (!frame) return;

    try {
      const cloned = frame.clone();
      const res = (await Promise.race([
        api.detect(Comlink.transfer(cloned, [cloned]), this.#threshold),
        cancel,
      ])) as Catalog.DetectionObjects | void;

      if (!res) return; // canceled
      this.objects.set(res);
      this.#track.writeJson(res);
      timeout = setTimeout(process, this.#interval);
    } catch (err) {
      // Swallow errors if shutting down
      if (await Promise.race([Promise.resolve(true), cancel])) return;
      console.error("detection detect error:", err);
    }
  };

  process();
});
js/hang/src/publish/audio/captions.ts (1)

96-96: Harden worklet loading with error handling.

addModule() can reject (network error, bad URL, blocked context). Catch and surface a user-friendly message; keep state consistent.

- await ctx.audioWorklet.addModule(CaptureWorklet);
+ try {
+   await ctx.audioWorklet.addModule(CaptureWorklet);
+ } catch (err) {
+   console.error("failed to load capture worklet:", err);
+   this.text.set(undefined);
+   return;
+ }
js/hang/src/publish/audio/index.ts (2)

135-135: Wrap addModule in try/catch to avoid silent failures.

- await context.audioWorklet.addModule(CaptureWorklet);
+ try {
+   await context.audioWorklet.addModule(CaptureWorklet);
+ } catch (err) {
+   console.error("failed to load capture worklet:", err);
+   return;
+ }

139-140: Guard channelCount for Firefox/unspecified values.

settings.channelCount can be undefined; passing undefined into AudioWorkletNodeOptions.channelCount and later arithmetic (bitrate) risks errors/NaN. Default to 1 when absent.

-        channelCount: settings.channelCount,
+        channelCount: settings.channelCount ?? 1,

And later in config:

-  numberOfChannels: u53(settings.channelCount),
-  bitrate: u53(settings.channelCount * 32_000),
+  numberOfChannels: u53((settings.channelCount ?? 1)),
+  bitrate: u53(((settings.channelCount ?? 1) * 32_000)),
js/hang/src/publish/audio/speaking.ts (1)

87-87: Add error handling around worklet registration.

- await ctx.audioWorklet.addModule(CaptureWorklet);
+ try {
+   await ctx.audioWorklet.addModule(CaptureWorklet);
+ } catch (err) {
+   console.error("failed to load capture worklet:", err);
+   return;
+ }
js/hang/src/watch/audio/index.ts (6)

10-12: Fix prop docs: Speaking/Captions take props, not booleans.

The comments suggest boolean flags, but the types are CaptionsProps and SpeakingProps. Adjust the docs to avoid misuse.

-	// Enable to download the speaking track. (boolean)
+	// Enable to download the speaking track.

Also applies to: 23-25


17-19: Align latency prop semantics with usage (ms) and wire it into AudioContext.

Comment says “latency hint for AudioContext” but code fixes it to "interactive" and only passes latency to the worklet. Either update docs or actually use it. Suggest using numeric latencyHint (seconds) derived from this.latency (ms).

-	// The latency hint to use for the AudioContext.
-	latency?: DOMHighResTimeStamp;
+	// Target end-to-end buffer latency in milliseconds (used by the render worklet and as AudioContext latencyHint).
+	latency?: DOMHighResTimeStamp;
@@
-		const context = new AudioContext({
-			latencyHint: "interactive",
+		const context = new AudioContext({
+			latencyHint: this.latency / 1000, // seconds
 			sampleRate,
 		});

Also applies to: 89-92


86-88: Comment is stale relative to code path.

We don't create an AudioContext when disabled. Update the comment to avoid confusion.

-		// NOTE: We still create an AudioContext even when muted.
-		// This way we can process the audio for visualizations.
+		// NOTE: We create an AudioContext only when enabled.
+		// Consumers can connect the node to speakers/analyzers as needed.

89-95: Expose the actual sampleRate via the signal.

#sampleRate is never set; expose context.sampleRate so readers don’t see undefined.

 		const context = new AudioContext({
 			latencyHint: this.latency / 1000, // seconds
 			sampleRate,
 		});
 		effect.cleanup(() => context.close());
+		effect.set(this.#sampleRate, context.sampleRate);

95-103: Specify AudioWorkletNode channel config to match the stream.

Without explicit options, the node defaults to stereo which can cause unwanted mixing when channelCount !== 2. Also clear the onmessage handler on cleanup.

-			// Create the worklet node
-			const worklet = new AudioWorkletNode(context, "render");
-			effect.cleanup(() => worklet.disconnect());
+			// Create the worklet node with explicit channel configuration
+			const worklet = new AudioWorkletNode(context, "render", {
+				numberOfInputs: 0,
+				numberOfOutputs: 1,
+				outputChannelCount: [channelCount],
+				channelCount,
+				channelCountMode: "explicit",
+				channelInterpretation: "discrete",
+			});
+			effect.cleanup(() => {
+				worklet.port.onmessage = null;
+				worklet.disconnect();
+			});

Optional: if you want processing to run even when nothing else connects, attach a silent sink:

const sink = context.createGain();
sink.gain.value = 0;
worklet.connect(sink).connect(context.destination);
effect.cleanup(() => sink.disconnect());

149-167: Add minimal backpressure to avoid unbounded decode queue if the context isn’t running.

If the node isn’t connected or the context is suspended, decodeQueueSize can grow. Add a soft cap.

-					decoder.decode(chunk);
+					decoder.decode(chunk);
+					// Soft backpressure: yield if we get too far ahead.
+					if (decoder.decodeQueueSize > 24) {
+						await new Promise((r) => setTimeout(r, 0));
+					}
📜 Review details

Configuration used: CodeRabbit UI

Review profile: CHILL

Plan: Pro

💡 Knowledge Base configuration:

  • MCP integration is disabled by default for public repositories
  • Jira integration is disabled by default for public repositories
  • Linear integration is disabled by default for public repositories

You can enable these sources in your CodeRabbit configuration.

📥 Commits

Reviewing files that changed from the base of the PR and between 59f5507 and 1c09a1e.

⛔ Files ignored due to path filters (1)
  • js/pnpm-lock.yaml is excluded by !**/pnpm-lock.yaml
📒 Files selected for processing (8)
  • js/hang/package.json (1 hunks)
  • js/hang/src/publish/audio/captions.ts (2 hunks)
  • js/hang/src/publish/audio/index.ts (2 hunks)
  • js/hang/src/publish/audio/speaking.ts (2 hunks)
  • js/hang/src/publish/video/detection.ts (2 hunks)
  • js/hang/src/util/hacks.ts (0 hunks)
  • js/hang/src/watch/audio/index.ts (3 hunks)
  • js/package.json (0 hunks)
💤 Files with no reviewable changes (2)
  • js/package.json
  • js/hang/src/util/hacks.ts
🧰 Additional context used
📓 Path-based instructions (2)
{rs,js}/hang/**

📄 CodeRabbit inference engine (CLAUDE.md)

All media logic belongs in the hang layer (rs/hang and js/hang)

Files:

  • js/hang/src/publish/video/detection.ts
  • js/hang/src/publish/audio/captions.ts
  • js/hang/src/publish/audio/index.ts
  • js/hang/src/publish/audio/speaking.ts
  • js/hang/src/watch/audio/index.ts
  • js/hang/package.json
js/**

📄 CodeRabbit inference engine (CLAUDE.md)

Place JS/TS packages under the js/ directory

Files:

  • js/hang/src/publish/video/detection.ts
  • js/hang/src/publish/audio/captions.ts
  • js/hang/src/publish/audio/index.ts
  • js/hang/src/publish/audio/speaking.ts
  • js/hang/src/watch/audio/index.ts
  • js/hang/package.json
🧠 Learnings (1)
📚 Learning: 2025-08-13T21:30:35.001Z
Learnt from: CR
PR: kixelated/moq#0
File: CLAUDE.md:0-0
Timestamp: 2025-08-13T21:30:35.001Z
Learning: Applies to js/pnpm-workspace.yaml : Use pnpm workspaces for JS/TS; manage package layout via js/pnpm-workspace.yaml

Applied to files:

  • js/hang/package.json
⏰ Context from checks skipped due to timeout of 90000ms. You can increase the timeout in your CodeRabbit configuration to a maximum of 15 minutes (900000ms). (1)
  • GitHub Check: Check
🔇 Additional comments (8)
js/hang/src/publish/video/detection.ts (1)

7-8: Ambient Module Declaration Exists – No Further Action Needed

The project already includes an ambient declaration for *?worker&url and the tsconfig covers it, so TypeScript will resolve these imports without errors.

• File js/hang/src/vite-env.d.ts declares:

// Add support for worklet imports
declare module "*?worker&url" {
  const workerUrl: string;
  export default workerUrl;
}

• js/hang/tsconfig.json extends the base and includes "src", ensuring the above .d.ts is picked up.

You can safely ignore the original comment—no additional types or tsconfig changes are required.

js/hang/src/publish/audio/captions.ts (1)

7-7: Vite-only import: verify TS support for ?worker&url.

Same concern as the video detection worker URL. Ensure vite/client types or an ambient module for *?worker&url exist to keep tsc --noEmit green.

Use the same script shared in detection.ts to verify.
Reference: Vite docs on worker URL query. (main.vitejs.dev)

js/hang/package.json (3)

41-42: Good: move to workspace:^ for local packages.

This aligns with pnpm/Yarn modern workspace resolution and the repo’s workspace setup.


45-46: Confirmed availability of @huggingface/transformers@^3.7.2
Ran npm view @huggingface/transformers version and npm view @huggingface/transformers versions --json | jq '.[-5:]', which returned version 3.7.2 as the latest release. Your lockfile should resolve to this version without issues.


43-43: Dependency range ^4.1.3 for Zod is valid

Verification shows that Zod v4.1.4 is published on npm, so the existing range resolves correctly:

  • npm view zod version → 4.1.4
  • npm pack zod@^4.1.3 succeeds, pulling v4.1.4

No change is strictly required; you can safely keep "zod": "^4.1.3".
If you’d like to pin to the exact latest minor, you can optionally bump to ^4.1.4, for example:

• js/hang/package.json (line 43)

-    "zod": "^4.1.3",
+    "zod": "^4.1.4",  // optional bump to include only subsequent patch releases
js/hang/src/publish/audio/index.ts (1)

14-17: Vite-only import for worklet: ensure repo-wide TS support.

As above, confirm ?worker&url typing. Also consider adding a short comment in code linking to Vite worker docs, since this is intentionally Vite-specific.

Reuse the typing check script. Reference: Vite worker URL query docs. (main.vitejs.dev)

js/hang/src/publish/audio/speaking.ts (1)

6-6: Vite-only worklet import: verify TS typing for ?worker&url.

Same action item as other files to keep tsc happy.

Use the typing verification script.
Reference: Vite docs for worker URL imports. (main.vitejs.dev)

js/hang/src/watch/audio/index.ts (1)

27-29: All worklet imports are consistent and no legacy loader remains
Every *.ts file under js/hang/src/watch and js/hang/src/publish uses the ?worker&url pattern, and there are no stray loadAudioWorklet references. Ready to merge.

@kixelated kixelated merged commit 97233e4 into main Aug 27, 2025
4 checks passed
@kixelated kixelated deleted the revert-webpack branch August 27, 2025 22:42
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants