Sending large amounts of data using TCPClient on Photon (revisited)

The Photon can stream out a whole lot of data using TCPClient, quite reliably, and pretty quickly, at least under ideal conditions. I had a done a test sending out 1 Mbyte of data, then closing and reopening the connection earlier, but the question of what happens if you do that over one connection, continuously, came up, so I tested that here.

I still get over 900 Kbytes/sec. Every byte sent out is accounted for (no dropped, added, or corrupted bytes), and you can do it without a single delay() call in the code.

The test ran for 9 hours and 44 minutes and 30,812 MB, yes, over 30 GB, was transferred without error. The connection closed at that point for unknown reasons, but a new connection was opened and things proceeded normally after that.

Of course this doesn’t prove that the Photon will always perform that well, it merely shows that it can. I will not be repeating this test with the Electron, unless someone wants to donate an unlimited SIM to me as I can think of better ways to spend US$30,000! LOL

// Test app for sending large amounts of data over a TCP connection

#include "Particle.h"

// This test can be run with system thread enabled or disabled. It's a little faster
// with it enabled, about 1150 Kbytes/sec. vs. about 900 Kbytes/sec with it disabled
// (the default).
// SYSTEM_THREAD(ENABLED);

// Retained memory is used to keep track of reboots.
// PHOTON ONLY: Connect VBAT to GND to the retained memory will be initialized on power-up if
// you are not using a coin cell battery
// Do NOT connect VBAT to GND on an Electron or you will short out the 3.3V regulator!
STARTUP(System.enableFeature(FEATURE_RETAINED_MEMORY));

// Finite state machine states
enum State { STATE_CONNECT, STATE_FILL_BUFFER, STATE_SEND, STATE_RETRY_WAIT };

// Keeps track of the number of times we've run setup(), which should be equal to the number
// of reboots since we don't use sleep modes in this test app
retained int setupCount = 0;

// Various constants

// bufSize is the number of bytes we make in a typical write call. Making this 2048 or
// larger can cause data corruption on the Photon. 1024 seems experimentally to be ideal;
// if you make it smaller it works without errors but the data rate drops.
const size_t bufSize = 1024;

// Various timeout values.
const unsigned long retryWaitTimeMs = 2000;
const unsigned long sendTimeoutMs = 60000;

// Set to the IP address of the server to connect to
IPAddress serverAddr(192,168,2,4);
const int serverPort = 7123;

// Global variables
State state = STATE_CONNECT;
TCPClient client;
unsigned long stateTime = 0;
unsigned char buf[bufSize];
size_t sentInBuf;
unsigned long totalSent = 0;
unsigned char bufStartChar;

void setup() {
	Serial.begin(9600);

	// Increment retained variable; should happen on every reboot (since we don't use sleep in
	// this program).
	setupCount++;
}

void loop() {
	switch(state) {
	case STATE_CONNECT:
		Serial.printlnf("** starting setupCount=%d millis=%lu", setupCount, millis());

		if (!client.connect(serverAddr, serverPort)) {
			// Connection failed
			stateTime = millis();
			state = STATE_RETRY_WAIT;
			break;
		}
		totalSent = 0;
		bufStartChar = (unsigned char) setupCount;
		state = STATE_FILL_BUFFER;
		// Fall through

	case STATE_FILL_BUFFER:
		// Each buffer has different bytes so we can detect corruption on the server
		for(size_t ii = 0; ii < bufSize; ii++) {
			buf[ii] = bufStartChar++;
		}
		sentInBuf = 0;

		state = STATE_SEND;
		stateTime = millis();
		// Fall through

	case STATE_SEND:
		if (client.connected()) {
			int count = client.write(&buf[sentInBuf], bufSize - sentInBuf);
			if (count == -16) {
				// Special case: Internal buffer is full, just retry at the same offset next time
				// I'm pretty sure the result code for this is different on the Core, and probably the Electron.

				if (millis() - stateTime > sendTimeoutMs) {
					// I never hit this debug statement in my tests
					Serial.printlnf("** timeout sending sentInBuf=%u totalSent=%lu millis=%lu", sentInBuf, totalSent, millis());
					client.stop();
					stateTime = millis();
					state = STATE_RETRY_WAIT;
				}
			}
			else
			if (count > 0) {
				// Normal case, sent some bytes. count may be less than the buffer size, which
				// means we need to send the rest later.
				stateTime = millis();
				totalSent += count;
				sentInBuf += count;

				if (sentInBuf >= bufSize) {
					// Sent whole buffer, refill next time
					state = STATE_FILL_BUFFER;
				}
			}
			else {
				// Error
				Serial.printlnf("** error sending error=%d sentInBuf=%u totalSent=%lu millis=%lu", count, sentInBuf, totalSent, millis());
				client.stop();
				stateTime = millis();
				state = STATE_RETRY_WAIT;
			}
		}
		else {
			Serial.printlnf("** connection closed totalSent=%lu millis=%lu", totalSent, millis());
			client.stop();
			stateTime = millis();
			state = STATE_RETRY_WAIT;
		}
		break;

	case STATE_RETRY_WAIT:
		if (millis() - stateTime > retryWaitTimeMs) {
			// Wait a few seconds before retrying
			state = STATE_CONNECT;
			break;
		}
		break;
	}
}


6 Likes

Hi @rickkas7

I have been playing around with the same question as you (how fast can the Photon stream TCP), and was getting similar results (about 1Mbit/sec).

Then I realized the limiting factor wasn’t the Photon at all, but my PC code!

Using your code as posted above, it is possible to get over 10Mbit/sec, apparently error-free.

In my case, a quick check with Wireshark showed that my receiving socket was experiencing TCP Zero Window problems. Fixed that and it jumped from 1 to 10Mbit/sec. Don’t know if that may be affecting your tests or not, but 10Mb/s is possible.

3 Likes

@OdLynx, can you share your improved PC code?

@bpr, I would share my code, if it were improved. Unfortunately, I misread rickkas7’ original post. I saw “bits” where he clearly said “bytes”. I can confirm his transfer rates, but can’t offer any improvement.

Changing my PC code only eliminated a problem and got me to where he already was. :grinning:

@rickkas7’s code can easily be adapted to read files from an SD card and send it to a TCP server. Here is a simple PHP script that can take this incoming data and store it in a file. To run this from the command prompt use:

php -q tcp_server.php

=====

#!/usr/bin/php -q
<?php
error_reporting(E_ALL);

/* Allow the script to hang around waiting for connections. */
set_time_limit(0);

/* Turn on implicit output flushing so we see what we're getting
 * as it comes in. */
ob_implicit_flush();

$address = '11.22.33.44'; // Your server's IP
$port = 10001; // Your port number

if (($sock = socket_create(AF_INET, SOCK_STREAM, SOL_TCP)) === false) {
    echo "socket_create() failed: reason: " . socket_strerror(socket_last_error()) . "\n";
}

if (socket_bind($sock, $address, $port) === false) {
    echo "socket_bind() failed: reason: " . socket_strerror(socket_last_error($sock)) . "\n";
}

if (socket_listen($sock, 5) === false) {
    echo "socket_listen() failed: reason: " . socket_strerror(socket_last_error($sock)) . "\n";
}
$dir = 'DataFiles';
// create new directory with 744 permissions if it does not exist yet
// owner will be the user/group the PHP script is run under
 if ( !file_exists($dir) ) {
     $oldmask = umask(0);  // helpful when used in linux server  
     mkdir ($dir,0744);
 }


echo getcwd() . "\n";
do {
    if (($msgsock = socket_accept($sock)) === false) {
        echo "socket_accept() failed: reason: " . socket_strerror(socket_last_error($sock)) . "\n";
        break;
    }
    /* Send instructions. */
    $msg = "\nWelcome to the PHP Test Server. \n" .
        "To quit, type 'quit'. To shut down the server type 'shutdown'.\n";
    socket_write($msgsock, $msg, strlen($msg));
    $newFile = strtotime(date('YmdHis')) . "_data.txt";
    echo $newFile;
    $fh = fopen($dir."//". $newFile, 'w');
    do {
        if (false === ($buf = socket_read($msgsock, 2048, PHP_NORMAL_READ))) {
            echo "socket_read() failed: reason: " . socket_strerror(socket_last_error($msgsock)) . "\n";
            break 2;
        }
        if (!$buf = trim($buf)) {
            continue;
        }
        if ($buf == 'quit') {
            break;
        }
        if ($buf == 'shutdown') {
            socket_close($msgsock);
            break 2;
        }
        $talkback = "PHP: You said '$buf'.\n";
        socket_write($msgsock, $talkback, strlen($talkback));
        echo "$buf\n";
        fwrite($fh, $buf . "\n"); 
    } while (true);
    socket_close($msgsock);
    fclose($fh);
} while (true);

socket_close($sock);
?>
1 Like

@rickkas7 - Am using your code as the base for sending data from a file on an SD card to a TCP server. It all works quite well EXCEPT that the last chunk of data goes missing when transferring files several MB (I have tried up to 4.1 MB). I am guessing somehow the last buffer does not get transmitted? I couldn’t find anything in the code that indicates this. Here is the modified code written as a function:

byte doTCPTransfer(){
  enum State { STATE_CONNECT, STATE_FILL_BUFFER, STATE_SEND, STATE_RETRY_WAIT };
  // Keeps track of the number of times we've run setup(), which should be equal to the number
  // of reboots since we don't use sleep modes in this test app
  int setupCount = 0;
  // Various constants

  // bufSize is the number of bytes we make in a typical write call. Making this 2048 or
  // larger can cause data corruption on the Photon. 1024 seems experimentally to be ideal;
  // if you make it smaller it works without errors but the data rate drops.
  const size_t bufSize = 512;

  // Various timeout values.
  const unsigned long retryWaitTimeMs = 2000;
  const unsigned long sendTimeoutMs = 60000;

  // Set to the IP address of the server to connect to
  IPAddress serverAddr(1,2,3,4);
  const int serverPort = 10001;

  // Global variables
  State state = STATE_CONNECT;
  TCPClient tcpclient;
  unsigned long stateTime = 0;
  unsigned char buf[bufSize];
  size_t sentInBuf;
  unsigned long totalSent = 0;
  unsigned char bufStartChar;

  // re-open the file for reading:
    if (!fh.open(fileName, O_READ)) {
  		Serial.println("data.txt not found for read");
      handleError("SD");
      return 0;
    }

    if(!fh)
    {
      Serial.println("SD open fail");
      return 0;
    }

    Serial.println("SD opened");
  //%%%%%%%%%%%%%%%%%%%%%%
  //Actual file transfer
        //Run this while loop as long as fh available
        while(fh.available())
        {
          //Here the switch snippet comes in
          //SWITCH SWITCH SWITCH SWITCH SWITCH SWITCH SWITCH
                      switch(state) {
                      case STATE_CONNECT:
                        Serial.printlnf("** starting setupCount=%d millis=%lu", setupCount, millis());

                        if (!tcpclient.connect(serverAddr, serverPort)) {
                          // Connection failed
                          stateTime = millis();
                          state = STATE_RETRY_WAIT;
                          break;
                        }
                        totalSent = 0;
                        bufStartChar = (unsigned char) setupCount;
                        Particle.process();
                        state = STATE_FILL_BUFFER;
                        // Fall through

                      case STATE_FILL_BUFFER:

                        Serial.printlnf("** filling buffer");
                        // Each buffer has different bytes so we can detect corruption on the server
                        for(size_t ii = 0; ii < bufSize; ii++) {
                          buf[ii] = fh.read();
                        }
                        sentInBuf = 0;
                        Particle.process();
                        state = STATE_SEND;
                        stateTime = millis();
                        // Fall through

                      case STATE_SEND:
                      Serial.printlnf("** sending buffer");
                        if (tcpclient.connected()) {
                          int count = tcpclient.write(&buf[sentInBuf], bufSize - sentInBuf);
                          if (count == -16) {
                            // Special case: Internal buffer is full, just retry at the same offset next time
                            // I'm pretty sure the result code for this is different on the Core, and probably the Electron.

                            if (millis() - stateTime > sendTimeoutMs) {
                              // I never hit this debug statement in my tests
                              Serial.printlnf("** timeout sending sentInBuf=%u totalSent=%lu millis=%lu", sentInBuf, totalSent, millis());
                              tcpclient.stop();
                              stateTime = millis();
                              state = STATE_RETRY_WAIT;
                            }
                          }
                          else
                          if (count > 0) {
                            // Normal case, sent some bytes. count may be less than the buffer size, which
                            // means we need to send the rest later.
                            stateTime = millis();
                            totalSent += count;
                            sentInBuf += count;

                            if (sentInBuf >= bufSize) {
                              // Sent whole buffer, refill next time
                              state = STATE_FILL_BUFFER;
                            }
                          }
                          else {
                            // Error
                            Serial.printlnf("** error sending error=%d sentInBuf=%u totalSent=%lu millis=%lu", count, sentInBuf, totalSent, millis());
                            tcpclient.stop();
                            stateTime = millis();
                            state = STATE_RETRY_WAIT;
                          }
                        }
                        else {
                          Serial.printlnf("** connection closed totalSent=%lu millis=%lu", totalSent, millis());
                          tcpclient.stop();
                          stateTime = millis();
                          state = STATE_RETRY_WAIT;
                        }
                        break;

                      case STATE_RETRY_WAIT:
                        if (millis() - stateTime > retryWaitTimeMs) {
                          // Wait a few seconds before retrying
                          state = STATE_CONNECT;
                          break;
                        }
                        break;
                      }
          //SWITCH SWITCH SWITCH SWITCH SWITCH SWITCH SWITCH
          //End of switch
        }
  fh.close();
  Serial.println("SD file closed");
  return 1;
  //%%%%%%%%%%%%%%%%%%%%%%
}

Right after posting I realized it could be related to the loop

while(fh.available){ //read, buffer & send }

As soon as the last buffer is read the while condition fails and therefore the loop ends resulting in the last buffer never getting transmitted!?? Could this be?

How can this be handled?

This is what is working now:

        while(!endwhile)
        {
          //Here the switch snippet comes in
          //SWITCH SWITCH SWITCH SWITCH SWITCH SWITCH SWITCH
                      switch(state) {
                      case STATE_CONNECT:
                        Serial.printlnf("** starting setupCount=%d millis=%lu", setupCount, millis());

                        if (!tcpclient.connect(serverAddr, serverPort)) {
                          // Connection failed
                          stateTime = millis();
                          state = STATE_RETRY_WAIT;
                          break;
                        }
                        totalSent = 0;
                        bufStartChar = (unsigned char) setupCount;
                        Particle.process();
                        state = STATE_FILL_BUFFER;
                        // Fall through

                      case STATE_FILL_BUFFER:

                        Serial.printlnf("** filling buffer");
                        // Each buffer has different bytes so we can detect corruption on the server
//===
                        toRead = 0;
                        numBytesAvailable = fh.available();
                        if (numBytesAvailable >= bufSize) {
                            toRead = bufSize;
                        } else {
                            toRead = numBytesAvailable;
                            breakwhile = 1;
                            Serial.println("##### setting breakwhile to 1");
                        }

                        for(size_t ii = 0; ii < toRead; ii++) {
                            buf[ii] = fh.read();
                        }
//===
//                        for(size_t ii = 0; ii < bufSize; ii++) {
//                          if(!fh.available()){
//                            breakwhile = 1;
//                            break;
//                          }
//                          buf[ii] = fh.read();
//                        }
                        sentInBuf = 0;
                        Particle.process();
                        state = STATE_SEND;
                        stateTime = millis();
                        // Fall through

                      case STATE_SEND:
                      Serial.printlnf("** sending buffer");
                        if (tcpclient.connected()) {
                          int count = tcpclient.write(&buf[sentInBuf], toRead - sentInBuf);
                          if (count == -16) {
                            // Special case: Internal buffer is full, just retry at the same offset next time
                            // I'm pretty sure the result code for this is different on the Core, and probably the Electron.

                            if (millis() - stateTime > sendTimeoutMs) {
                              // I never hit this debug statement in my tests
                              Serial.printlnf("** timeout sending sentInBuf=%u totalSent=%lu millis=%lu", sentInBuf, totalSent, millis());
                              tcpclient.stop();
                              stateTime = millis();
                              state = STATE_RETRY_WAIT;
                            }
                          }
                          else
                          if (count > 0) {
                            // Normal case, sent some bytes. count may be less than the buffer size, which
                            // means we need to send the rest later.
                            stateTime = millis();
                            totalSent += count;
                            sentInBuf += count;

                            if (sentInBuf >= bufSize) {
                              // Sent whole buffer, refill next time
                              state = STATE_FILL_BUFFER;
                            }
                            if (breakwhile == 1){
                              Serial.println("##### setting endwhile to 1");
                              endwhile = 1;
                              tcpclient.stop();
                              break;
                            }
                          }
                          else {
                            // Error
                            Serial.printlnf("** error sending error=%d sentInBuf=%u totalSent=%lu millis=%lu", count, sentInBuf, totalSent, millis());
                            tcpclient.stop();
                            stateTime = millis();
                            state = STATE_RETRY_WAIT;
                          }
                        }
                        else {
                          Serial.printlnf("** connection closed totalSent=%lu millis=%lu", totalSent, millis());
                          tcpclient.stop();
                          stateTime = millis();
                          state = STATE_RETRY_WAIT;
                        }
                        break;

                      case STATE_RETRY_WAIT:
                        if (millis() - stateTime > retryWaitTimeMs) {
                          // Wait a few seconds before retrying
                          state = STATE_CONNECT;
                          break;
                        }
                        break;
                      }
          //SWITCH SWITCH SWITCH SWITCH SWITCH SWITCH SWITCH
          //End of switch
        }

@rickkas7 - unable to follow the logic of various if else conditions in the switch so sort of bypassed some of them. There seems to be no clean exit on successful transfer so built then in by calling the tcpclient.close() in one of the switch cases.

Yes, that should work. The reason that code doesn’t have an exit path for when you’ve sent a certain number of bytes is that it was for testing very large amounts of data, basically as much data as can be sent before the connection resets. There’s another example that restricts the data size, but what you did should work fine.