Streaming audio over UDP to VLC media player

Hello,

I am trying to get my photon to stream audio data over UDP or similar. I think a good solution is to stream over UDP to VLC media player but cannot get this to work.

I have followed this example :https://github.com/dmiddlecamp/particle-photon-audio-stream

Which allowed me to stream the audio and save the files onto my laptop. I now want (close to) live or ‘real-time’ playback. I have modified the audio-stream main file to:

#include "SparkIntervalTimer.h"
#include "SimpleRingBuffer.h"
#include <math.h>

#define MICROPHONE_PIN A0
#define AUDIO_BUFFER_MAX 8192

#define SINGLE_PACKET_MIN 512
#define SINGLE_PACKET_MAX 1024

#define SERIAL_DEBUG_ON true


#define AUDIO_TIMING_VAL 125 /* 8,000 hz */


uint8_t txBuffer[SINGLE_PACKET_MAX + 1];


SimpleRingBuffer audio_buffer;
//SimpleRingBuffer recv_buffer;

UDP Udp;
unsigned int localPort = 8888;

// version without timers
unsigned long lastRead = micros();
unsigned long lastSend = millis();
char myIpAddress[24];

IntervalTimer readMicTimer;

//float _volumeRatio = 0.50;
int _sendBufferLength = 0;
unsigned int lastPublished = 0;
bool _isRecording = false;

volatile int counter = 0;


void setup() {
    digitalWrite(D2, HIGH);
    #if SERIAL_DEBUG_ON
    Serial.begin(115200);
    #endif

    setADCSampleTime(ADC_SampleTime_3Cycles);

    pinMode(MICROPHONE_PIN, INPUT);
    pinMode(D7, OUTPUT);


    int mySampleRate = AUDIO_TIMING_VAL;

    Particle.variable("ipAddress", myIpAddress, STRING);
    Particle.variable("sample_rate", &mySampleRate, INT);
    Particle.publish("sample_rate", " my sample rate is: " + String(AUDIO_TIMING_VAL));

    IPAddress myIp = WiFi.localIP();
    sprintf(myIpAddress, "%d.%d.%d.%d", myIp[0], myIp[1], myIp[2], myIp[3]);

    audio_buffer.init(AUDIO_BUFFER_MAX);

    lastRead = micros();

    readMicTimer.begin(readMic, AUDIO_TIMING_VAL, uSec);

}

unsigned int lastLog = 0;
unsigned int lastClientCheck = 0;


void loop() {
    unsigned int now = millis();

    if ((now - lastClientCheck) > 100) {
        lastClientCheck = now;

        checkClient = audioServer.available();
        if (checkClient.connected()) {
            audioClient = checkClient;
        }
    }


    #if SERIAL_DEBUG_ON
    if ((now - lastLog) > 1000) {
        lastLog = now;

        Serial.println("counter was " + String(counter));
        //Serial.println("audio buffer size is now " + String(audio_buffer.getSize()));
        counter = 0;

    }

    #endif

    sendEvery(100);

}

// Callback for Timer 1
void readMic(void) {
    uint16_t value = analogRead(MICROPHONE_PIN);

    value = map(value, 0, 4095, 0, 255);
    audio_buffer.put(value);


    counter++;
}




void startRecording() {
    if (!_isRecording) {
        readMicTimer.begin(readMic, AUDIO_TIMING_VAL, uSec);
    }
    _isRecording = true;
}

void stopRecording() {
    if (_isRecording) {
        readMicTimer.end();
    }
    _isRecording = false;
}

void sendEvery(int delay) {
    // if it's been longer than 100ms since our last broadcast, then broadcast.
    if ((millis() - lastSend) >= delay) {
        sendAudio();
        lastSend = millis();
    }
}
// Callback for Timer 1
void sendAudio(void) {
    int count = 0;
    int storedSoundBytes = audio_buffer.getSize();
    
    while (count < storedSoundBytes) {

        if (audio_buffer.getSize() < SINGLE_PACKET_MIN) {
            break;
        }
        
        int size = min(audio_buffer.getSize(), SINGLE_PACKET_MAX);
        int c = 0;
        for(int c = 0; c < size; c++) {
            txBuffer[c] = audio_buffer.get();
        }
        count += size;


        // send it!
        Udp.sendPacket(txBuffer, size, IPAddress(192,168,0,198),localPort);
        //audioClient.write(txBuffer, size);
    }
}

Running this and then opening the stream on VLC results in:
main error: connection error: Connection timed out

ftp error: connection failed

main debug: no access modules matched

main debug: dead input

qt debug: IM: Deleting the input

main debug: changing item without a request (current 16/17)

main debug: nothing to play

Can anyone point me in right direction? Sorry this is my first project of this kind and I’m struggling with it.

Hi,

I don’t have experience with these kinds of things on the particle, but I design audio systems for a living in my day job.

Is there a reason you want this to work via UDP? Have you tried the vanilla TCP build as offered at that link you shared?

If you want close to real time connection speeds, modifying your buffer and packet structure would be a better way to go. The only reason the UDP is used in VOIP is for simplicity, not for better reliability or speed (UDP has no error correction of any sort).

You post has given me an idea of something I need to try… I’ll let you know if I get the particle working with a gstreamer AES67 library. That would have a shot at what you need for latency and sound quality.

Welcome to the wacky world of audio.
bob

1 Like

Hi Bob,
Thanks for your response. No particular reason for UDP, I thought that it was better protocol than TCP. I’ll try to get it working on TCP first. Using the vanilla code I am still unable to get the output.

Running the code on particle I try to connect using VLC player to rtp://@1234 I get a connection but no playback.

main debug: `rtp://@:1234’ successfully opened

lua debug: Trying Lua scripts in C:\Users\user\AppData\Roaming\vlc\lua\meta\art

lua debug: Trying Lua scripts in C:\Program Files\VideoLAN\VLC\lua\meta\art

lua debug: Trying Lua playlist script C:\Program Files\VideoLAN\VLC\lua\meta\art\00_musicbrainz.luac

lua debug: Trying Lua playlist script C:\Program Files\VideoLAN\VLC\lua\meta\art\01_googleimage.luac

lua debug: Trying Lua playlist script C:\Program Files\VideoLAN\VLC\lua\meta\art\02_frenchtv.luac

lua debug: Trying Lua playlist script C:\Program Files\VideoLAN\VLC\lua\meta\art\03_lastfm.luac

main debug: no art finder modules matched

qt debug: IM: Setting an input

Am I messing up my VLC settings? I’m guessing as I make the connection on VLC there is a stream and I am managing to connect to it. :thinking::face_with_raised_eyebrow:

Thanks for your response anyway, let me know if you manage to get gstreamer working.

Nice.

Unless you are going to have 100’s of devices on the same network (or audiophile bitrates), TCP is the way to go here.

I’m confused about your lua settings - aren’t those typically just cosmetic, not functional?

If you are having trouble with VLC, have you considered using the server.js script from that repo with max/msp for audio debugging?

If you aren’t familiar with max/msp, you should grab yourself the free trial right away. It supports node.js and many commercial companies use it to create standalone applications, including in the security camera market. It’s incredibly user friendly, and has amazing real-time streaming capabilities. Using MAX 8 with the asio driver on windows is crucial to dialing in your buffer and packet sizes. https://cycling74.com/

1 Like

One more thing - gstreamer tools are great for audio, but there are some issues with gstreamer for video.

If you are just doing audio, MAX is the way to go.

If you need 10-20 microsecond latency for a commercial project, AES67 is gonna be your best bet (AVB is a mess). However, for nearly 100% of other applications, simple TCP and 20ms latency is perfectly acceptable.

1 Like

Okay thanks for your advice. I’ll have a look at Max now. I’ve used the server.js script with success. Just can’t get it to stream the output instead of saving it to file.

I understand that this line

var outStream = fs.createWriteStream(“test.wav”);

Writes to test.wav but I have never done js and struggling to modify this so that I can hear it real-time.

I’ve heard good things about howler https://howlerjs.com/ if you can’t get the audio working with Max. (Max is largely designed for realtime playback).

1 Like

Okay. So I have installed Max and trying to run the server.js on there.
I get the error “jsui: server.js: Javascript TypeError: fs.createWriteStream is not a function, line 29”

I guess because there is no access to files from Max. How can I change this to an output on Max rather than a file?

Thanks for your advice, if I’m able to get it streaming on Max looks like it has lots of other useful files I can use.

As for protocol, TCP seems like the right way to go for now to get proof of concept. If you are able to get AES67 working though would be great to have Photon using that protcol.

I believe the reason UDP did not work is that VLC can’t play raw UDP frames like it can over TCP. It only handles RTP (Real-time Transport Protocol) frames with UDP transport. You’d have to generate the RTP and RTSP frames and that’s probably more work than you want to deal with.

Okay Rickkas,

Thanks for your advice. I’ll have a read up, must admit I did think it wouldn’t be so much to get started haha. I feel like I’m down this rabbithole now though! Should I be able to get VLC working if I revert back to TCP?

As far as the UDP vs. TCP debate – I worked at Lucent when a lot of the VoIP stuff was first coming out, and remember the discussions. It wasn’t because UDP was “easier,” but rather because the error-correction in TCP really was a detractor for real-time communication. When you are streaming content across the network, the last thing you would want is your network stack holding up your stream because one or two packets came out of order or needed to be re-transmitted. TCP sends an ACK for the traffic as well as reassembles all the packets in the correct order before it delivers it up the OSI stack to your running application. If you are streaming audio, one packet can be lost and ignored without really destroying the stream. TCP also adds additional overhead (that used to be a concern when edge ports were running at 10MB/s or even slower) compared to UDP.

3 Likes