Sending large data off Photon using TCPClient


#1

I was wondering exactly how fast you can get data off a Photon using a direct TCP connection (TCPClient), and how reliable is it?

Well, after tweaking the code and some parameters, in my somewhat arbitrary test of opening a TCP connection and sending 1 Mbyte of data up to a local server, then closing the connection, I got an average of 805 Kbytes/sec. 1.3 seconds to upload 1 MB of data.

I repeated this test continuously, uploading almost 4 GB of data in 86 minutes, and there were no failures. Not a byte out of place, no failed connections, no errors on either side.

Bytes per upload 1048576
Number of uploads 3986
Minimum time 1091 ms, rate = 961.1 Kbytes/sec
Maximum time 3253 ms, rate = 322.3 Kbytes/sec
Mean time 1302 ms, rate = 805.4 Kbytes/sec
Test time 5210 sec

Presumably you won’t be capturing and uploading 1 MB of data every couple seconds on a Photon, but theoretically it could keep up with it, with a sufficiently fast Wi-Fi and server!

// Test code for sending large amounts of data off a Photon

// 1024 seems to be a good size for this buffer. Making it 2048 or larger tends to cause
// errors writing. Making it 512 works fine, but will limit the maximum transfer rate.
// I get a maximum transfer rate of around 499 Kbytes/sec with a 512 byte buffer.
// It's around 870 Kbytes/sec with a 1024 byte buffer.
// If you're short on RAM and don't need the extra speed, using a smaller buffer is fine.
byte buffer[1024];
int bufOffset = 0;
long dataSize = 1024 * 1024;
long dataSent = 0;
byte startByte = 0;
byte nextByte = 0;

TCPClient client;
byte server[] = { 192, 168, 2, 4 };
int port = 8123;


enum { STATE_CONNECT, STATE_FILL_BUFFER, STATE_WRITE };

int state = STATE_CONNECT;

void setup() {
    Serial.begin(9600);
}

void loop() {
    switch(state) {
      case STATE_CONNECT:
        if (client.connect(server, port)) {
            dataSent = 0;
            nextByte = startByte++;
            state = STATE_FILL_BUFFER;
        }
        else {
            Serial.println("ERROR failed to connect");
            delay(5000);
        }
        break;

      case STATE_FILL_BUFFER:
        for(int ii = 0; ii < sizeof(buffer); ii++) {
            buffer[ii] = nextByte++;
        }
        bufOffset = 0;
        state = STATE_WRITE;
        // Fall through

      case STATE_WRITE:
        int requestSize = sizeof(buffer) - bufOffset;

        int count = client.write(&buffer[bufOffset], requestSize);
        if (count == -16) {
            // This seems like a buffer full error of some sort. This is not fatal,
            // and after a few cycles you'll be able to write again.
            // There is a possibility that there is a different error code for this on the Core!

            // Serial.printlnf("Got a -16 at %d", bufOffset);

            // You don't have to delay at all here; it will just keep retrying. But
            // because the transmit buffer is full, you won't starve the connection
            // for data with a short delay here.
            delay(10);
        }
        else
        if (count < 0) {
            Serial.printlnf("ERROR write failed %d", count);
            client.stop();
            state = STATE_CONNECT;
            delay(1000);
        }
        else {
            // Note: client.write returns the number of bytes written. It may be
            // smaller than the buffer size, and this is not fatal. Just write the
            // rest in another write call!
            bufOffset += count;
            if (bufOffset >= sizeof(buffer)) {
                // Done sending this buffer
                dataSent += sizeof(buffer);
                if (dataSent >= dataSize) {
                    // Sent all of the buffers for this connection
                    Serial.printlnf("SUCCESS %ld bytes sent", dataSent);
                    client.stop();
                    state = STATE_CONNECT;
                }
                else {
                    // Load the buffer with data again
                    state = STATE_FILL_BUFFER;
                }
          }
          else {
            //Serial.printlnf("sent count=%d", count);
          }
        }
        break;
    }
}


Sending large amounts of data using TCPClient on Photon (revisited)
Electron TCPClient.write issue
TCPClient and NodeJS server
Max TCP Buffer size
Need help Sending jpeg/bytes to the Webserver with Linksprite serial ttl camera
TCPServer() usage and bandwidth issue
How to send large file from Particle to server?
Name resolving of TCPClient when connecting to other Photon running mDNS
Photon WiFi Throughput
MQTT publish rate of Photon
Sending high frequency/large data packets via particle (BCM chip)
Electron tcpclient.write issues
UDP problem 0.4.9
#2

Can this be used to load data onto a Photon’s memory or a SD card connected to the Photon?

I’m actually looking for a way to load / update data on the P1 modules extra 1MB of memory where I was planning on storing LCD screen bitmap data. But I guess a Photon + SD card would work also.


#3

Of course yes, after writing some software. I was going to test the download direction but I think that’s less common. I’ll do it after my theoretically Photon-compatible micro SD card socket breakout board arrives next week.


#4

Sweet!

So can this can be used remotely right? It’s not just for local connections is it? Sorry if that is a stupid question.

@peekay123 how does this differ from your FTP large data transfer code?


#5

Yes, the server doesn’t need to be local, though then the maximum transfer rate will bump up against the limits of slower Internet connections for upload with other traffic. Probably not for download to Photon, since most connections, at least in the US, have faster download and slower upload.


#6

Sounds great.

I’ll wait to see how your download to the Photon goes :smiley:

Here is the code @peekay123 is working on for downloding to Photon via FTP server if your interested.


#7

A quick (and perhaps stupid) question here:

What needs to be running on the server? The IP address of the server in this sketch is 192.168.2.4:8123

Thanks


#8

It depends. This actual post had a Java server that monitors both sides and does a bunch of error detection, but that’s unnecessarily complicated for normal use.

Here’s an example of a server written in node.js that receives audio from a Photon:

And here’s a more generic one for receiving data: