Can't get good quality with gstreamer due to fragmenting issue

[skirsten]

So I don't know what I am doing wrong. But the visual and audio quality is just much worse than if I stream the same pipeline to a file and play it back.

via MoQ:

gst-launch-1.0 -v -e filesrc location=sample.mp4 ! qtdemux name=demux \
  demux.video_0 ! decodebin ! queue ! identity sync=true ! nvh264enc bitrate=6000 gop-size=120 rc-mode=cbr-ld-hq zerolatency=true ! h264parse ! queue ! identity sync=true check-imperfect-timestamp=true check-imperfect-offset=true ! isofmp4mux name=mux chunk-duration=1 fragment-duration=1 ! hangsink url="***" broadcast="***" \
  demux.audio_0 ! decodebin ! queue ! identity sync=true ! voaacenc bitrate=160000 ! aacparse ! queue ! identity sync=true check-imperfect-timestamp=true check-imperfect-offset=true ! mux.

to file:

gst-launch-1.0 -v -e filesrc location=sample.mp4 ! qtdemux name=demux \
  demux.video_0 ! decodebin ! queue ! identity sync=true ! nvh264enc bitrate=6000 gop-size=120 rc-mode=cbr-ld-hq zerolatency=true ! h264parse ! queue ! identity sync=true check-imperfect-timestamp=true check-imperfect-offset=true ! mp4mux name=mux ! filesink location=output.mp4 \
  demux.audio_0 ! decodebin ! queue ! identity sync=true ! voaacenc bitrate=160000 ! aacparse ! queue ! identity sync=true check-imperfect-timestamp=true check-imperfect-offset=true ! mux.

Also looking at the audio waveform, it looks like its outputting mono 😕.

Edit:

So I inspected the output file output.fmp4 (using isofmp4mux name=mux chunk-duration=1 fragment-duration=1 ! filesink location=output.fmp4) and it has way too many keyframes (which would explain the worse quality). But if I use chunk-duration=2000000000 fragment-duration=2000000000 which should mean 2 second chunks (matching the 60fps and 120 GoP) I am getting this error:

thread '' panicked at hang-gst/src/sink/imp.rs:146:38:
failed to parse: Mp4(DuplicateBox(trun))

When I pipe it to a file instead, the output looks much better (and the keyframes make sense).

What is this "TRUN" box and how can I work around it or do you have any idea what to do here?

[kixelated]

No clue why you're getting worse quality. MoQ only performs networking, so it has to be something in your gstreamer pipeline.

A fragment/chunk size of 1 is currently required otherwise fMP4 starts adding latency. I'd like to get rid of the fMP4 conversion (like the sink) but it requires a more complicated gstreamer plugin that I don't really feel like figuring out.

[kixelated]

Also there's a recent mp4-box PR that would fix your trun issue, at least for parsing: https://github.com/kixelated/mp4-atom/pull/49/files

[skirsten]

I will check out that PR, for now I managed to figure this out: isofmp4mux name=mux chunk-duration=1 fragment-duration=2000000000 which seems to work with hangsink while at the same time only doing a keyframe every two seconds.

I don't know enough about gstreamer to understand why, but to me it seems that the fragment-duration=1 is forcing constant keyframe generation, which then eats into the bitrate and results in worse quality.

But now with chunk-duration=1 fragment-duration=2000000000 the audio works for 2 seconds and then is silent for 2 seconds 😮‍💨. I guess this has nothing to do with hang, but if you have any ideas please let me know

[skirsten]

Hmm, the audio issue is not there when using the file output. So I guess its a issue in the browser 🤔 ?

Looking at the audio waveform (top file, bottom playback in chrome via hang) it's much quieter (volume is set to 1), looks like mono and has visible distortions / crackling.
Could this be that the browser is doing some filtering, thinking its supposed to be human voices :)

[skirsten]

audio: {
  latency: 2100,
},

Fixes the gaps and crackling at least :) but of course that will add 2 extra seconds of latency - unusable 😮‍💨

[skirsten]

Okay, I can reproduce the mono issue. If I add audioconvert mix-matrix="<<(float)0.0, (float)0.0>, <(float)0.0, (float)1.0>>" to only play audio on the right speaker, there is no sound.

Meaning the browser is only playing the left channel and mixing that to stereo. I tried to find the issue in the code, but no luck yet. All seems to make sense 🤔

Edit: I can't figure it out 😮‍💨 Nothing is working consistently and I can't make sense of it. Giving up for now...

[skirsten]

After the stereo audio fix I switched to ffmpeg which works a whole lot better. There is still some audio crackling on startup.
Maybe during catchup related to this:
https://github.com/kixelated/moq/blob/04613432bde845bac2b57ddd6957984d0af3e8e1/js/hang/src/watch/audio/index.ts#L196

But at least with ffmpeg the video is exactly as I encode it and nothing is messing with the latency, keyframes or fragmentation.

Here are the flags that I am currently using:

  -r 60 -g 120 \
  -c:v h264_nvenc -b:v 6000k -preset:v llhq -tune ull -zerolatency 1 -strict_gop 1 \
  -c:a libopus -b:a 160k -vbr constrained \
  -f mp4 -movflags cmaf+separate_moof+delay_moov+skip_trailer+frag_every_frame \

Closing for now, as it seems to be mostly a gstreamer / encoding problem.