Echo cancellation

[RASPIAUDIO]

I made it work on the ESP32 S3 Muse Radio but it’s picking up it’s own response, I think webrtc has built-in noise cancellation but how to activate it?

https://www.youtube.com/watch?v=Q9BVurtly_M&embeds_referring_euri=https%3A%2F%2Fcommunity.openai.com%2F&source_ve_path=Mjg2NjY

[juberti-oai]

There's no echo cancellation built into ESP32, you'll need to find an external AEC library like https://www.espressif.com/en/solutions/audio-solutions/esp-afe.

[shenxiaozheng]

We can refer to these examples to implement the echo cancellation functionality. https://github.com/espressif/esp-skainet/tree/master/examples

[nischalj10]

hey all, I tried echo cancellation (AEC) via ESP-AFE but its too slow for realtime needs. I think the esp32s3 chipset is not capable of doing wifi+realtime audio+ noise cancellation all at the same time.

were any of you successful in accomplishing noise cancellation?

[RASPIAUDIO]

hey all, I tried echo cancellation (AEC) via ESP-AFE but its too slow for realtime needs. I think the esp32s3 chipset is not capable of doing wifi+realtime audio+ noise cancellation all at the same time.

were any of you successful in accomplishing noise cancellation?

@nischalj10 can you share your code ? I can try to run AEC on one core and webrtc on the other

[shenxiaozheng]

I have tested the AEC algorithm of ESP before. CPU usage is less than 20% of a single core. Do you have any tests in this area.

[nischalj10]

@nischalj10 can you share your code ? I can try to run AEC on one core and webrtc on the other

hey @RASPIAUDIO - i have done a simple from scratch implementation of just aec (without realtime webrtc) on my setup and that also does not seem to work. here's the project.

here's my hardware details -

• seeed xiao esp32s3
• inmp 441 mems mic
• max98357a amp for audio out

could you try to run this and mod it so it works?

[nischalj10]

I have tested the AEC algorithm of ESP before. CPU usage is less than 20% of a single core. Do you have any tests in this area.

hey @shenxiaozheng, is it possble to share your implementation, maybe i have missed something here?

[shenxiaozheng]

https://github.com/nischalj10/aec-test/blob/main/main/aec_handler.h
You can modify the configuration of the above file to this way。
// Voice communication specific settings
afe_config.aec_init = true;
afe_config.se_init = false;
afe_config.voice_communication_init = false;
afe_config.wakenet_init = false;
afe_config.vad_init = false;

[nischalj10]

hey @shenxiaozheng , thanks for the suggestion!
the afe_config.voice_communication_init = false; nudged me to use the SR mode rather than VC mode I was previously using.

i have updated the code and it works now. will try to port the configuration of this test repo to my original codebase with openai-realtime and see how it does.

[shenxiaozheng]

@nischalj10 Yes, using SR is correct. Looking forward to your great work.

[nischalj10]

update : as i try to do the port, i notice a fundamental issue. the opus encoder expects 20ms (BUFFER_SIZE=320) chunks of audio, while aec expects 32ms (BUFFER_SIZE=512) - and we can't change this for either of them.

we will need to maintain ring buffers to get this working properly. have been unsuccessful today in getting it to work.

[shenxiaozheng]

@nischalj10 Yes, the frame length of the encoding and decoding algorithm is not consistent with that of the AEC algorithm. By increasing the buffer, they can work together.

[PoohWoah]

@nischalj10 Yes, the frame length of the encoding and decoding algorithm is not consistent with that of the AEC algorithm. By increasing the buffer, they can work together.是的，编解码算法的帧长与AEC算法的帧长不一致。通过增加缓冲区，它们可以一起工作。

Hello, I used the awakened SR to do echo cancellation, but when the distance between the speaker and the microphone is relatively close, there will still be a situation of asking yourself and answering yourself, how to optimize it?

[PoohWoah]

@nischalj10 Yes, the frame length of the encoding and decoding algorithm is not consistent with that of the AEC algorithm. By increasing the buffer, they can work together.是的，编解码算法的帧长与AEC算法的帧长不一致。通过增加缓冲区，它们可以一起工作。

Yue Xin official seems to support OPUS codec, maybe use this will be useful? Have you tried it?

[shenxiaozheng]

@PoohWoah

1. 建议喇叭离麦克风越远越好，这样可以保证麦克风录音不截幅
2. 建议使用谐波失真较小的喇叭
3. 如果cpu剩余较多，建议使用vc，打开aec和nsnet，，aec进行回声消除，nsnet是神经网络的噪声抑制，参考https://github.com/espressif/esp-skainet/blob/master/examples/voice_communication/main/main.c

[PoohWoah]

@PoohWoah

1. 建议喇叭离麦克风越远越好，这样可以保证麦克风录音不截幅
2. 建议使用谐波失真较小的喇叭
3. 如果cpu剩余较多，建议使用vc，打开aec和nsnet，，aec进行回声消除，nsnet是神经网络的噪声抑制，参考https://github.com/espressif/esp-skainet/blob/master/examples/voice_communication/main/main.c

好的谢谢，目前我只加入了aec并且测试了其是否生效，nsnet还没使用，我会在年后进行一个测试，nsnet支持adf框架吗？

[nischalj10]

@PoohWoah , 您能否分享一下您如何处理 opus 编码的 320 缓冲区长度和 AEC 的 512 缓冲区长度之间的不匹配。我目前创建了两个单独的任务来执行此操作 - 一个将麦克风数据输入 AEC，另一个从中获取。

[PoohWoah]

我很乐意，但是我目前已经在假期了。恐怕需要等我过完春节才可以发你，不过我具体代码可能帮不到你太多，或许对你有用。

…

---Original--- From: "Nischal ***@***.***> Date: Fri, Jan 24, 2025 18:27 PM To: ***@***.***>; Cc: ***@***.******@***.***>; Subject: Re: [openai/openai-realtime-embedded-sdk] Echo cancellation (Issue#10) @PoohWoah , 您能否分享一下您如何处理 opus 编码的 320 缓冲区长度和 AEC 的 512 缓冲区长度之间的不匹配。我目前创建了两个单独的任务来执行此操作 - 一个将麦克风数据输入 AEC，另一个从中获取。 — Reply to this email directly, view it on GitHub, or unsubscribe. You are receiving this because you were mentioned.Message ID: ***@***.***>

[nischalj10]

我很乐意，但是我目前已经在假期了。恐怕需要等我过完春节才可以发你，不过我具体代码可能帮不到你太多，或许对你有用。
…

谢谢你！