Issue with mixed generated noise output after mixed mp3 stops playing

[level0dev]

As a prefix, I'm new to this library, ESP32 in general and even coding with C, so please excuse my inexperience. This might be an expected behavior, but I want to check before I try to solve the issue on my own.

I'm working on a project where I would add "radio noise" to mp3 files stored on SD card, my project is not too dissimilar from @alastaira's London Calling Prop. The project uses the A1S audiokit board.

My current approach is to generate static noise and mix it with an mp3 file stored on SD card. I achieved this with the below code based on this discussion.

The code mixes noise with the mp3 file as expected, but when the mp3 file has finished playing the noise turns into a clicking sound. I would expect the noise to keep playing as normal after the file has finished.

Is this clicking sound/change to the noise output an expected behavior or a bug?

• if a bug I'm happy to open an issue.
• If expected, how can I keep the generated noise playing normally after the mp3 stops playing?

#include <SPI.h>
#include <SD.h>
#include "AudioTools.h"
#include "AudioTools/AudioLibs/AudioBoardStream.h"
#include "AudioTools/AudioCodecs/CodecMP3Helix.h"


const int chipSelect=PIN_AUDIO_KIT_SD_CARD_CS;
File audioFile;
AudioBoardStream i2s(AudioKitEs8388V1); // final output of decoded stream
InputMixer<int16_t> mixer;

EncodedAudioStream decoder(&i2s, new MP3DecoderHelix());

PinkNoiseGenerator<int16_t> Noise;
GeneratedSoundStream<int16_t> NoiseIn(Noise);

VolumeStream volumeSong(decoder);
VolumeStream volumeNoise(NoiseIn);

StreamCopy copier(i2s, mixer);

// Arduino Setup
void setup(void) {
  Serial.begin(115200);
  while (!Serial)
    ;
  
  auto config = i2s.defaultConfig(TX_MODE);
  config.sd_active = true;
  i2s.begin(config);

  SD.begin(chipSelect);
  audioFile = SD.open("/test2.mp3");
  
  if (!audioFile) {
    Serial.println("Failed to open MP3 file!");
    stop();
    return;
  } else {
    Serial.println("Opened the MP3 file");
  }
  Serial.println("Card Mount succeeded");

  NoiseIn.begin(config);
  Noise.begin();

  volumeSong.setVolume(0.8);
  volumeSong.begin();
  volumeNoise.setVolume(0.3);
  volumeNoise.begin();
  decoder.setStream(audioFile);
  decoder.transformationReader().setResultQueueFactor(14);
  decoder.begin();

  mixer.add(volumeSong);
  mixer.add(volumeNoise);
  mixer.begin(config);
  copier.begin();
}

void loop() {
  copier.copy();
}

[pschatzmann]

Please follow the advice given here

[level0dev]

Thanks - so it's an expected behavior due to the mixer expecting equal amount of data from each of its sources.

What is the proper way to dynamically change the amount of sources a mixer is expecting?

I would like to change what the mixer is expecting each time a file finishes, so the mixer plays the remaining source(s) correctly. Then, when a new mp3 starts playing, I would change it back to expect the extra source.

Is simply calling mixer.remove() and mixer.add() when the mp3 stops/starts the answer?

[level0dev]

Well, calling mixer.remove() doesn't have any affect and mixer.remove(VolumeSong) (the mp3 stream I add in setup), seems to stop the whole playback after a single clicking sound (so it doesn't seem to change the speed the noise is played back to normal).

I would love an example showing how to tweak the mixer on the fly to keep the sound playing at normal speed after the mp3 has finished.

[pschatzmann]

In my suggestion above, I was recommending to use the OutputMixer which gives you more control and flexibility.
mixer.remove() should work for the stream components that report available() == 0

Can you provide a short example that demonstrates your issue ?

[level0dev]

So, since my last message I managed to get it working by calling mixer.remove(0). Using the stream index was the only thing that would play the noise at normal speed after the mp3 has finished.

However, there is an audible gap in audio stream when the mp3 finishes using that method. I was expecting the mixing to be seamless - once again, I'm not sure if that's expected behavior or a bug?

I'll experiment with the OutputMixer to see if I'm running into similar issues.

---

Anyway, here's the code without the index in mixer.remove() which still doesn't work as expected:

#include <SPI.h>
#include <SD.h>
#include "AudioTools.h"
#include "AudioTools/AudioLibs/AudioBoardStream.h"
#include "AudioTools/AudioCodecs/CodecMP3Helix.h"


const int chipSelect=PIN_AUDIO_KIT_SD_CARD_CS;
File audioFile;
AudioBoardStream i2s(AudioKitEs8388V1); // final output of decoded stream
InputMixer<int16_t> mixer;

bool audioFileIsPlaying = false;

EncodedAudioStream decoder(&i2s, new MP3DecoderHelix());

PinkNoiseGenerator<int16_t> Noise;
GeneratedSoundStream<int16_t> NoiseIn(Noise);

VolumeStream volumeSong(decoder);
VolumeStream volumeNoise(NoiseIn);

StreamCopy copier(i2s, mixer);

// Arduino Setup
void setup(void) {
  Serial.begin(115200);
  while (!Serial)
    ;
  
  auto config = i2s.defaultConfig(TX_MODE);
  config.sd_active = true;
  i2s.begin(config);

  SD.begin(chipSelect);
  audioFile = SD.open("/test2.mp3");
  
  if (!audioFile) {
    Serial.println("Failed to open MP3 file!");
    stop();
    return;
  } else {
    Serial.println("Opened the MP3 file");
  }
  Serial.println("Card Mount succeeded");

  NoiseIn.begin(config);
  Noise.begin();


  volumeSong.setVolume(0.8);
  volumeSong.begin();
  volumeNoise.setVolume(0.3);
  volumeNoise.begin();
  decoder.setStream(audioFile);
  decoder.transformationReader().setResultQueueFactor(14);
  decoder.begin();

  mixer.add(volumeSong);
  mixer.add(volumeNoise);
  mixer.begin(config);
  copier.begin();
  audioFileIsPlaying = true;
}

void loop() {
  copier.copy();

  if (audioFileIsPlaying && !audioFile.available()) {
    audioFileIsPlaying = false;
    mixer.remove();
    Serial.println("MP3 playback finished, removed from mixer.");
}

}

Sorry for the messy code - it's a testing grounds while I'm trying to learn so I'm not paying attention to organizing it. I'm also happy to provide the test2.mp3 if needed - it's the winamp demo sound in 124kbps, 44.100kHz and stereo.

[pschatzmann]

Did you try something like the following:

void setup() {
   mixer.setLimitToAvailableData(true); // no output when file has ended
}

void loop() {
  if (!copier.copy()){
    mixer.remove();
    Serial.println("MP3 playback finished, removed from mixer.");
  }
}

[level0dev]

When I try the above code it still creates the clicking sound. It doesn't seem to matter where in setup I add the mixer.setLimitToAvailableData(true); (just before I begin the mixer or on the very beginning of setup).

Just to be sure, I have also tested it with yours and mine IF functions in loop - both create the clicking when calling mixer.remove(); without the index.

As such, I can confirm that this method does not fix the issue.

Also - can you please explain how does the if (!copier.copy()) work? Does it return false when the stream is changed? Does the copier stop itself when the stream is changed (hence returning false) then restarts?

If the latter, that would explain the gap I hear when I remove the 0-index stream from the mixer.

[pschatzmann]

setLimitToAvailableData(true) makes sure that the smallest common available data is mixed. So at the end of file copier.copy returns 0 bytes: an int value of 0 maps to false and any other value to true. Removing the empty stream should make sure that data is getting available again.

• The clicking could come from the small delay added by the processing: you could try to increase the I2S buffer to prevent any underflow
• Or it could come from the recalculated weights which make your noise much louder: you could try to replace the volumeSong at index 0 with an instance of NullStream instead of removing it. This should not change any weights.

[level0dev]

I just want to be sure we are not misunderstanding one another:

• When I mention the "clicking sound," I'm talking about the generated noise sped up due to the mixer only receiving half the data
• The gap I'm hearing is more of a pop (I'll call it a "pop" from now on to avoid confusion) and is a distinct issue from the clicking mentioned above.

I'll try both methods. I believe the larger buffer method was in the original code I used, so I'll simply add it back in.

However, I don't know the syntax to replace the stream with NullStream - ' volumeSong.setStream(NullStream);' throws an compilation error.

Finally, is there a reason why simply calling mixer.remove(); doesn't work with my code? Is it my code or a bug?

[level0dev]

I figured out that NullStream is a class (I think that's the right term in C nomenclature) and created a new var of that class, started it with .begin() and put that into the VolumeSong stream.

That seems to have solved the problem partially - there's less of a pop sound (which probably was due to volume/weight change), but there's still an audible gap.

The "buffer increase" in original was added to the decoder via decoder.transformationReader().setResultQueueFactor(14); - I don't think this will affect anything if I'm adding silence using NullStream. I think the buffer needs increasing on the mixer itself, but again, my inexperience with C and difficulty grokking the documentation is stopping me in my tracks. The only thing in examples I found was about creating different types of buffers.

Mixer doesn't have .transformationReader() method, so I could use some help on how to increase the buffer size in my example code?