Sending live audio broadcast from INMP441 to MQTT Server.

[umairz1]

Hello,
I have been looking through the arduino audio tools library that you have created and tried quite a few examples here to understand its functionality. Wonderful work done here might I add. I wanted to ask if I could use the library tools to broadcast live audio from my inmp441 mic to my mqtt server. I am using ESP32 Wroom 32 board.
I basically want to receive this audio from anywhere in the world on my smartphone app. I have sent raw pcm data through mqtt's client.publish function and the raw bytes are received, but they either cannot be played or are too noisy to understand. I even started from local communication using udp and faced the same issue. I have tried the streams-i2s-webserver-wav example which works perfectely on local network, I just want to be able to access it on a remote network too.
I have read the wiki and tried to modify the code to send to mqtt but to no avail. I also did a search for similar discussions and you said to use the ip communication example streamcopy method but I didnt understand how it worked. Maybe you can guide me in the write direction and tell if is it even possible I will be really thankful.

[pschatzmann]

I dont have any experience with mqtt but I would not see any issues. Mqtt clients usually implement the stream api, so you can just use it as copy target. Just make sure that you stay within the limits if the protocol.

I suggest to encode it as wav, keep the channels at 1 and the sample rate low e.g at 8000, so that you know what you get on the target.

Maybe you just overlooked the issues with this microphone?

[umairz1]

The microphone works in itself that if I record in wav format locally it gives a crystal clear audio when played. Even your stream i2s webserver example outputs easy to distinguish audio on the ip. Just that when I publish on mqtt the audio breaks or is not played. Maybe due to network latency or something, but if that was the issue it would'nt have given the same issue on udp.

Can you explain what you mean by copy target. Is is the copier.copy() function? How do I set mqtt as the target?

And also if you know any other communication method through which I can try other than mqtt I'll be more than happy to try that too.

[pschatzmann]

I think you dismissed my answer too early: know your hardware and do not ignore my documentation!

• https://github.com/pschatzmann/arduino-audio-tools/wiki/It's-not-working
• https://github.com/pschatzmann/arduino-audio-tools/wiki/External-ADC

I think you just did not do your homework.

I can't provide any MQTT support and only you know which of the many libraries you are using. But you should be able to figure this one out yourself. Just try to understand the design of it by looking at your header files and documentation.

[umairz1]

Okay I will look into this. Thank you for the reply.

[pschatzmann]

I added a simple example that sends white noise using the Arduino ArduinoMqttClient library.

Adapting this to your library and to a different audio source should be trivial

[umairz1]

Thank you for providing the example. I really appreciate it. I got it working with my mic source and it sends the wav audio to my mqtt client after some time. But atleast the audio is intelligible unlike before. If you don't mind can I paste the code if any other person wants to try:

/**
 * @file communication-mqtt-send.ino
 * @author Phil Schatzmann
 * @brief Send Audio File to MQTT.
 * I am using the ArduinoMQTTClient library from Arduino provided by the library manager.
 * We can just copy the audio data to the MQTT client!
 * @version 0.1
 * @date 2023-09-07
 * 
 * @copyright Copyright (c) 2023
 */

#include "WiFi.h"
#include "ArduinoMqttClient.h"
#include "AudioTools.h"

#define SIZE 1024
#define N 256

// Communication
const char* ssid = "SSID";    // your network SSID (name)
const char* password = "PWD";    // your network password (use for WPA, or use as key for WEP)
const char* broker = "BROKER_IP"; 
const char* topic  = "MQTT_TOPIC";
int        port     = 1883;
WiFiClient wifiClient;
MqttClient mqttClient(wifiClient);

// Audio
AudioInfo info(8000, 1, 16);
I2SStream i2s;                                                    
EncodedAudioStream out_stream(&mqttClient, new WAVEncoder());     // encode as wav file
StreamCopy copier(out_stream, i2s, SIZE);                                            // copies sound to MQTT client
auto config = i2s.defaultConfig(RX_MODE);

// Connect to Wifi
void connectWIFI() {
  // attempt to connect to WiFi network:
  Serial.print("Attempting to connect to WPA SSID: ");
  Serial.println(ssid);
  WiFi.begin(ssid, password);

  Serial.print("Connecting to WiFi ..");
  while (WiFi.status() != WL_CONNECTED) {
    Serial.print('.');
    delay(1000);
  }

  Serial.println("You're connected to the network");
  Serial.println();  
}

// Connect to MQTT Server
void connectMQTT() {
  // You can provide a unique client ID, if not set the library uses Arduino-millis()
  // Each client must have a unique client ID
  mqttClient.setId("AudioTools");

  // You can provide a username and password for authentication
  // mqttClient.setUsernamePassword("username", "password");

  Serial.print("Attempting to connect to the MQTT broker: ");
  Serial.println(broker);

  if (!mqttClient.connect(broker, port)) {
    Serial.print("MQTT connection failed! Error code = ");
    Serial.println(mqttClient.connectError());

    stop();
  }

  Serial.println("You're connected to the MQTT broker!");
  Serial.println();
}  

// Send audio to MQTT Server
void sendMQTT() {
    // make sure that we write wav header
    out_stream.begin(info);

    // send message, the Print interface can be used to set the message contents
    mqttClient.beginMessage(topic, SIZE * N, true);

    // copy audio data to mqtt: 100 * 1024 bytes
    copier.copyN(N);

    mqttClient.endMessage();
}


void setup() {
  // Initialize logger 
  Serial.begin(115200);
  AudioLogger::instance().begin(Serial, AudioLogger::Info);

  // connect
  connectWIFI();
  connectMQTT();

  // setup i2s mic
  Serial.println("starting I2S...");
  config.copyFrom(info);
  config.i2s_format = I2S_STD_FORMAT;
  config.is_master = true;
  config.use_apll = true;
  i2s.begin(config);
  Serial.println("I2S started...");

  // send Audio
  sendMQTT();
}

void loop() {
  // call poll() regularly to allow the library to send MQTT keep alives which
  // avoids being disconnected by the broker
  mqttClient.poll();
  delay(1000);
}

This one is working for me with the inmp441 mic.

[umairz1]

Also I thought I should try copying audio from mqtt client as source and a speaker connected with a max98357a i2s amplifier. I used the same mqtt library for ease of use (I was using the pubsubclient library before) and configured the speaker like the "communication-ip-receive.ino" example.
On the client side, I had a python script that sends mic audio using pyaudio to the mqtt broker which I then receive on the esp32 by subscribing to topic. I have been able to receive audio in raw form and in wav format in the mqtt callback function successfully. But I wasnt able to successfully copy the audio to my speaker. Once or twice I heard a little crackling in the start and then it goes silent. Fiddling with the buffersize on both ends helped but no audible sound was played.
Here is my attempt at the code:

/**
 * @file communication-mqtt-send.ino
 * @author Phil Schatzmann
 * @brief Send Audio File to MQTT.
 * I am using the ArduinoMQTTClient library from Arduino provided by the library manager.
 * We can just copy the audio data to the MQTT client!
 * @version 0.1
 * @date 2023-09-07
 * 
 * @copyright Copyright (c) 2023
 */

#include "AudioTools.h"
#include "ArduinoMqttClient.h"
#include <WiFi.h>

#define SIZE 1024
#define N 256

// Communication
const char* ssid = "SSID";    // your network SSID (name)
const char* password = "PWD";    // your network password (use for WPA, or use as key for WEP)
const char* broker = "BROKER_IP"; 
const char* topic  = "TOPIC";
int        port     = 1883;
WiFiClient wifiClient;
MqttClient mqttClient(wifiClient);

//Audio Stuff
AudioInfo info(8000, 1, 16);
I2SStream out; 
MeasuringStream out_stream(out);
StreamCopy copier(out, out_stream, SIZE);     
auto config = out.defaultConfig(TX_MODE);

void connectWIFI() {
  // attempt to connect to WiFi network:
  Serial.print("Attempting to connect to WPA SSID: ");
  Serial.println(ssid);
  WiFi.begin(ssid, password);

  Serial.print("Connecting to WiFi ..");
  while (WiFi.status() != WL_CONNECTED) {
    Serial.print('.');
    delay(1000);
  }

  Serial.println("You're connected to the network");
  Serial.println();  
}

void connectMQTT() {
  // You can provide a unique client ID, if not set the library uses Arduino-millis()
  // Each client must have a unique client ID
  mqttClient.setId("AudioTools");

  // You can provide a username and password for authentication
  // mqttClient.setUsernamePassword("username", "password");

  Serial.print("Attempting to connect to the MQTT broker: ");
  Serial.println(broker);

  if (!mqttClient.connect(broker, port)) {
    Serial.println("MQTT connection failed! Error code = ");
    Serial.println(mqttClient.connectError());

    while(1);
  }

  Serial.println("You're connected to the MQTT broker!");
  Serial.println();
}  

void setup() {
  Serial.begin(115200);
  AudioLogger::instance().begin(Serial, AudioLogger::Info);

  connectWIFI();
  connectMQTT();

  //set the message receive callback
  mqttClient.onMessage(onMqttMessage);
  Serial.print("Subscribing to topic: ");
  Serial.println(topic);
  Serial.println();

  mqttClient.subscribe(topic, 1);

  // start I2S
  Serial.println("starting I2S...");
  config.copyFrom(info); 
  config.buffer_size = 1024;
  config.buffer_count = 6;
  out.begin(config);
  Serial.println("I2S started...");
}

void loop() { 
  mqttClient.poll();

  if (!mqttClient.connected()){
    connectMQTT();
  }
}

//MQTT Callback (to receive audio from mqtt)
void onMqttMessage(int messageSize) {
  // we received a message, print out the topic and contents
  Serial.println("Received a message with topic '");
  Serial.print(mqttClient.messageTopic());
  Serial.print("', duplicate = ");
  Serial.print(mqttClient.messageDup() ? "true" : "false");
  Serial.print(", QoS = ");
  Serial.print(mqttClient.messageQoS());
  Serial.print(", retained = ");
  Serial.print(mqttClient.messageRetain() ? "true" : "false");
  Serial.print("', length ");
  Serial.print(messageSize);
  Serial.println(" bytes:");

  //use the Stream interface to print the contents
  while (mqttClient.available()) {
    Serial.print((char)mqttClient.read());
  }
  Serial.println("");

  //Copy data to speaker
  copier.copyN(N);
}

[pschatzmann]

Your mqttClient.read() will completely mess up the audio data. After all that's what the copier.copy() is supposed to do.

I would not recommend to process the playing of the audio in the callback, but rather move it to the loop and process it with individual copy() commands there...

[umairz1]

I have commented our the mqttClient.read() function and moved the copier in the loop but I am getting no output. In the serial monitor it shows this error:

[I] I2SConfig.h : 151 - pin_data: 22
I2S started...
E (6959) I2S: i2s_read(2264): RX mode is not enabled
[E] I2SESP32.h : 109 - size_t audio_tools::I2SDriverESP32::readBytes(void*, size_t)
[I] StreamCopy.h : 139 - StreamCopy::copy 1024 -> 0 -> 0 bytes - in 0 hops
E (6980) I2S: i2s_read(2264): RX mode is not enabled
[E] I2SESP32.h : 109 - size_t audio_tools::I2SDriverESP32::readBytes(void*, size_t)
[I] StreamCopy.h : 139 - StreamCopy::copy 1024 -> 0 -> 0 bytes - in 0 hops
E (7000) I2S: i2s_read(2264): RX mode is not enabled
[E] I2SESP32.h : 109 - size_t audio_tools::I2SDriverESP32::readBytes(void*, size_t)
[I] StreamCopy.h : 139 - StreamCopy::copy 1024 -> 0 -> 0 bytes - in 0 hops

[pschatzmann]

So, why are you reading from I2S ?
This does not make any sense.

Assuming it is not a wav file, you are supposed to copy from mqttClient to out or a MultiOutput if you want to provide the data to multiple outputs.
I am also not sure if your onMqttMessage is even necessary: Did you try w/o it ?
In this case the poll will also not be necessary any more.

[umairz1]

Yes it should be from the mqttclient. Poor work on my part. MeasuringStream out_stream(mqttClient); works but the sound is crackling with no intelligible audio.

[umairz1]

Also the audio is playing in chunks and gets cut off before any word is completed. But I can tell the person on the mic end is talking.

[umairz1]

So, why are you reading from I2S ? This does not make any sense.

Assuming it is not a wav file, you are supposed to copy from mqttClient to out or a MultiOutput if you want to provide the data to multiple outputs. I am also not sure if your onMqttMessage is even necessary: Did you try w/o it ? In this case the poll will also not be necessary any more.

Yes you are right, the callback function is not needed for copier to work. The poll is required to keep the broker connected though. Since it is familiar with the client.loop() function of the pubsubclient mqtt library I was previously working with.
I tried with copying the raw pcm audio sent to the esp32 using the measuringstream(mqttclient) but it breaks in between words since chunks are being sent and copied. I tried fiddling with the chunk size (on the python side) and the greater the size the longer the audio is played but it takes a while to fill the buffer and then send to the esp and play. I tried to send wav audio to my esp and like the new WAVEncoder() function, used the decoder one to decode and play but to no avail.

//Audio Stuff
AudioInfo info(8000, 1, 16);
I2SStream out; 
MeasuringStream out_mqtt(mqttClient);
EncodedAudioStream out_stream(&out_mqtt, new WAVDecoder());
StreamCopy copier(out, out_stream, SIZE);     
auto config = out.defaultConfig(TX_MODE);

[pschatzmann]

But that's completely wrong!
I suggest that you go back to the documentation and read it again.

Copy needs to be from the MQTTClient to the EncodedAudioStream which has I2SStream as output!
Or Copy from the MQTTClient to MeasuringStream which has EncodedAudioStream as output which has I2SStream as output!

[umairz1]

Like this?

I2SStream out; 

EncodedAudioStream wavedecode(&out, new WAVDecoder());
MeasuringStream measure_stream(wavedecode);
StreamCopy copier(measure_stream, mqttClient, SIZE);

//////OR//////

EncodedAudioStream out_stream(&out, new WAVDecoder());
StreamCopy copier(out_stream, mqttClient, SIZE);

[umairz1]

I2S started...
[I] StreamCopy.h : 305 - try write - 2 (open 1024 bytes) 
[I] StreamCopy.h : 305 - try write - 3 (open 1024 bytes) 
[I] StreamCopy.h : 305 - try write - 4 (open 1024 bytes) 
[I] StreamCopy.h : 305 - try write - 5 (open 1024 bytes) 
[I] StreamCopy.h : 305 - try write - 6 (open 1024 bytes) 
[I] StreamCopy.h : 305 - try write - 7 (open 1024 bytes) 
[I] StreamCopy.h : 305 - try write - 19 (open 1024 bytes) 
[I] StreamCopy.h : 305 - try write - 20 (open 1024 bytes) 
[I] StreamCopy.h : 305 - try write - 21 (open 1024 bytes) 
[I] AudioStreams.h : 1109 - ==> Bytes per second: 0
[E] StreamCopy.h : 299 - write to target has failed! (1024 bytes)
[I] StreamCopy.h : 139 - StreamCopy::copy 1024 -> 1024 -> 0 bytes - in 22 hops

This is the result in both copy methods.

[pschatzmann]

I guess you just forgot to call begin() on any of these classes...

[umairz1]

Yes that was embarrasing. I got the stream working but it still is in little chunks with a lot of noise but no intelligible voice audio.
Here is the audiologger output: (I first tested with 8000 rate, but 16k and 8k gave the same results)

[I] CodecWAV.h : 393 - isValid: true
[I] I2SStream.h : 74 - virtual void audio_tools::I2SStream::setAudioInfo(audio_tools::AudioInfo)
[I] AudioTypes.h : 88 - sample_rate: 16000
[I] AudioTypes.h : 89 - channels: 1
[I] AudioTypes.h : 90 - bits_per_sample: 16
[I] CodecWAV.h : 396 - WAVDecoder writing first sound data
[I] StreamCopy.h : 305 - try write - 2 (open 0 bytes) 
[I] StreamCopy.h : 139 - StreamCopy::copy 1024 -> 1024 -> 1024 bytes - in 2 hops
[I] StreamCopy.h : 139 - StreamCopy::copy 1024 -> 1024 -> 1024 bytes - in 1 hops
[I] StreamCopy.h : 139 - StreamCopy::copy 1024 -> 1024 -> 1024 bytes - in 1 hops
[I] StreamCopy.h : 139 - StreamCopy::copy 1024 -> 1024 -> 1024 bytes - in 1 hops
[I] StreamCopy.h : 139 - StreamCopy::copy 1024 -> 1024 -> 1024 bytes - in 1 hops
[I] StreamCopy.h : 139 - StreamCopy::copy 1024 -> 1024 -> 1024 bytes - in 1 hops
[I] StreamCopy.h : 139 - StreamCopy::copy 1024 -> 1024 -> 1024 bytes - in 1 hops
[I] StreamCopy.h : 139 - StreamCopy::copy 1024 -> 1024 -> 1024 bytes - in 1 hops
[I] StreamCopy.h : 139 - StreamCopy::copy 44 -> 44 -> 44 bytes - in 1 hops

[umairz1]

So, this one worked for me on the speaker. After some testing, I have found that it works best with min delay on 8000 sampling rate and chunk size 32768 on the python side where I send wav audio from mic. Also it works better when I use the local mqtt broker server on my pc instead of the remote VPS server where the mosquitto mqtt broker was originally installed. The audio is played after 4-5 secs after sending from mic.

/**
 * @file communication-mqtt-send.ino
 * @author Phil Schatzmann
 * @brief Send Audio File to MQTT.
 * I am using the ArduinoMQTTClient library from Arduino provided by the library manager.
 * We can just copy the audio data to the MQTT client!
 * @version 0.1
 * @date 2023-09-07
 * 
 * @copyright Copyright (c) 2023
 */

#include "AudioTools.h"
#include "ArduinoMqttClient.h"
#include <WiFi.h>

#define SIZE 1024
#define N 256

// Communication
const char* ssid = "SSID";    // your network SSID (name)
const char* password = "PWD";    // your network password (use for WPA, or use as key for WEP)
const char* broker = "BROKER_IP"; 
const char* topic  = "TOPIC";
int        port     = 1883;
WiFiClient wifiClient;
MqttClient mqttClient(wifiClient);

//Audio Stuff
AudioInfo info(8000, 1, 16);
I2SStream out; 

FilteredStream<int16_t, double> inFiltered(out, 1);
EncodedAudioStream out_stream(&inFiltered, new WAVDecoder());
MeasuringStream measure_stream(out_stream);
StreamCopy copier(measure_stream, mqttClient, SIZE);

auto config = out.defaultConfig(TX_MODE);

void connectWIFI() {
  // attempt to connect to WiFi network:
  Serial.print("Attempting to connect to WPA SSID: ");
  Serial.println(ssid);
  WiFi.begin(ssid, password);

  Serial.print("Connecting to WiFi ..");
  while (WiFi.status() != WL_CONNECTED) {
    Serial.print('.');
    delay(1000);
  }

  Serial.println("You're connected to the network");
  Serial.println();  
}

void connectMQTT() {
  // You can provide a unique client ID, if not set the library uses Arduino-millis()
  // Each client must have a unique client ID
  mqttClient.setId("AudioTools");

  // You can provide a username and password for authentication
  // mqttClient.setUsernamePassword("username", "password");

  Serial.print("Attempting to connect to the MQTT broker: ");
  Serial.println(broker);

  if (!mqttClient.connect(broker, port)) {
    Serial.println("MQTT connection failed! Error code = ");
    Serial.println(mqttClient.connectError());

    while(1);
  }

  Serial.println("You're connected to the MQTT broker!");
  Serial.println();
}  

void setup() {
  Serial.begin(115200);
  AudioLogger::instance().begin(Serial, AudioLogger::Info);

  connectWIFI();
  connectMQTT();

  Serial.print("Subscribing to topic: ");
  Serial.println(topic);
  Serial.println();

  mqttClient.subscribe(topic, 1 );  //qos 1

  // start I2S
  Serial.println("starting I2S...");
  config.copyFrom(info); 
  config.buffer_size = 1024;
  config.buffer_count = 6;
  out.begin(config);
  Serial.println("I2S started...");
}

void loop() { 
  out_stream.begin(info);

  if (!mqttClient.connected()){
     connectMQTT();
   }
   else{
     //Copy data to speaker
     copier.copyN(N);
     mqttClient.poll();
   }
}

[pschatzmann]

Your logic works only correctly when you know the data size in advance.

So your issue actually comes from ignoring my documentation: https://github.com/pschatzmann/arduino-audio-tools/wiki/Introduction#additional-processing-in-the-loop

I would suggest to call poll only when a copy fails: if (copier.copy()==0) mqttClient.poll(); You then need to adjust the logic around out_stream.begin as well

[umairz1]

Hmm this makes sense. I admit I have read parts of the documentation that I thought were according to my use case but looks like I still have a lot to unpack. I appreciate your guidance and patience in helping me address these issues. I am making the suggested changes and currently I am also trying to integrate both the mic and speaker code into one, to get two way communication going with the client and controller end. I will let you know how it goes. If I have any further questions or run into any issues during the process, I'll reach out for your assistance.

Thank you again for your support.

[pschatzmann]

I have never looked at the performance and overhead in using MQTT for audio. You might need to have a closer look at this.
In the worst case you will need to save the audio to a file first (or PSRAM) first before transmitting it over a slow connection.

[Hanseman66]

Hi Guys

This subject was interesting .
But i did not find the right hardware ..

But i did have a ESP32-A1S dev kit.

Can i use it as well ?

I looked around here on all of your work .
But i did not really get it ( I'm still new with Arduino )
So i would be glad if u could help me out :D

Br
Hans

[pschatzmann]

I am not sure what you mean with A1S Dev Kit and from where you want to get the audio.

If your board has a codec chip you need to use this method described in the Wiki
.

The AudioKit and LyraT have proper 16 bit stereo support: so it will be even easier....

[Hanseman66]

Hi

That is exactly the board i have on my desk at home.
And it fun to get to work as described in the post's.

Br,
Hans

[Hanseman66]

HI

I tryed with some of the examples.
But i always get :
Compilation error: AudioLibs/I2SCodecStream.h: No such file or directory

But the file is there .
in :
C:\Sync\SynologyDrive\Arduino\libraries\audio-tools\src\AudioTools\AudioLibs

What am i missing ?

Br
H

[pschatzmann]

I guess you forgot to install the audio-driver library ?

[Hanseman66]

Hi
I have installed this on
https://github.com/pschatzmann/arduino-audio-driver
Is there another ?
Br
H