TCPServer() usage and bandwidth issue

Hi,

I am using TCPServer() abstraction with Photon to send the around 448 bytes of data to the connected client at every 64ms. But when I verify all the packets on the client side, I found few packets were missed.

And after debugging I found the loss happens on Server side!! Is there any limitation of TCPServer() implementation on data rate?

Please help me to resolve this.

Thanks,
Dhaval

Hi,

I have done one more experiment. Run the server on the PC and tried to send the data to the same client at the same frequency i.e. 448 bytes at every 64ms. But there was no packet loss.

So it seems there is something wrong with TCPServer() abstraction or I am not using it correctly!!

Please help me to resolve this.

Thanks,
Dhaval

1 Like

Can someone please help me here?

Yes there is a limitation due to limited default RX buffer size of the device.

If your code does not ensure to keep the buffer below a certain high water mark, your buffer might wrap round overwriting not yet read data.
This has nothing to do with the TCP “packet safety”, since you are not actually loosing packets, but incoming packets are just overwriting older packets payloads in the buffer.

I have not had the need to look into that, but I’d assume there is a way to increase the buffer size (either via a not yet documented method overload or at least via local build of the system open source firmware).
But usually streamlining the application firmware to deal with that situation should be the first approach.

Hi @ScruffR,

Thanks for your reply!

I am not receiving any data on TCP, I am just sending data over TCP to the connected client. So how does the RX buffer size affecting it? And what is the hight water mark value for the buffer?

I have to send the data at the specified rate, so that can’t be change. So I think, I have to increase the buffer size. How can I achieve that?

Thanks,
Dhaval

1 Like

Ah, sorry - misinterpreted the communication direction.

In this case my question would be, if you really are loosing packets or are they just never sent in the first place?
Keeping a precise timing is not always easy with WiFi and cloud involved.

You could go with SYSTEM_MODE(SEMI_AUTOMATIC) to keep the cloud out of the equation and if you could provide some stripped down code that shows your problem it would be easier to judge too.

About the buffer size I’ve just found this, which seems a bit small :wink:
https://github.com/spark/firmware/blob/develop/wiring/inc/spark_wiring_tcpclient.h

#define TCPCLIENT_BUF_MAX_SIZE	128

uint8_t _buffer[TCPCLIENT_BUF_MAX_SIZE];

Hi @ScruffR,

I think the packets were never sent over the network and we have verified this using Wireshark packet capture. Ans also there is not scope of loosing packets over the network with TCP!

I am sending the packets in the local network, so I think cloud and probably WiFi should not be the issue. So do you still think that SYSTEM_MODE(SEMI_AUTOMATIC) will help?

Let me explain the implementation:

  1. Capture ADC channels in timer interrupt at every 4ms and accumulate that data in buffer (448 bytes).
  2. When buffer is full send the data over TCP to the connected client.
  3. While sending data over TCP keep capturing in alternate buffer, to prevent packet loss. So two buffers are used in alternate fashion.

The buffer max size, defined in the header, is for TCP Client and I am using TCP Server.

Thanks,
Dhaval

If you are running default SYSTEM_MODE(AUTOMATIC) the cloud will add some latency between iterations of loop(). This should be in the 1ms range, but depending on your global location it could be more.
But for tracking down the problem removing any unnecessary "cycle hogs" might help.

Yes, but the communication inside the TCPServer class is performed by use of its private TCPClient _client field

https://github.com/spark/firmware/blob/develop/wiring/src/spark_wiring_tcpserver.cpp

size_t TCPServer::write(const uint8_t *buffer, size_t size)
{
    return _client.write(buffer, size);
}

Hi @ScruffR,

I have tried to increase the TCPCLIENT_BUF_MAX_SIZE to 2048, but still observe the packet loss!!
Also tried after adding SYSTEM_MODE(AUTOMATIC), but no improvement.

What is restricting to send the data even in the local network?

Thanks,
Dhaval

Actually I meant you might consider not to use AUTOMATIC since it might contribute to your problem. Try SEMI_AUTOMATIC instead.

By mistake I mentioned AUTOMATIC, actually I used SEMI_AUTOMATIC. But I didn't see any improvement. Also after doing that I face some difficulties to re-program new application using Web IDE.

@dhaval, if you do a search, you’ll find creative ways to deal with OTA with the Photon in SEMI_AUTOMATIC mode.

@peekay123,
That is fine… But actually it is not resolving the problem of packet loss at Server end!!

@ScruffR, I have a similar configuration and I can confirm the buffer overwriting. I am reading a couple of megabytes of data from a photon using the following code:

	void readFromPhoton(String file){
	auto bytesRead =0;
	uint8_t buf[512]; 
	do{
		bytesRead = fh->readFile(file, buf, sizeof(buf)); // Read file from SD Card
		//!--> dserver->write(buf, sizeof(buf)); 
		for(auto i = 0; i< bytesRead; i++) dserver->write(buf[i]); //!--> Ok, no buffer overwrite
	}while( (bytesRead == sizeof(buf)) && bytesRead > 0); // continue while still reading chunks of sizeof(buffer)from card..
	dclient.stop();
	fh->flush();
}

Here, I’m outputting the contents of a file on a SDCard (using SDFatLib) to the client. calling write(uint8) while usable, is still extremely slow (about 15kB/s). While the SDCard is at SPI_HALF_SPEED, I see a major improvement by using write(const uint8_t*, size_t) block write (275kB/s on very large >50MB files).

Now here’s the catch: the write(uint8) correctly transfers the files over to the client, although veeeery slow. The block write results in incomplete transfered files.Every here and there, part of the file is missing.

How would one go about solving this? Has anybody else had any experience with this?

I remember @rickkas7 whipping up a nice bulk transfer sketch. Maybe he can chime in here.

or see

1 Like

Yes, you must use a block write with a buffer and length or it will be very, very slow. The optimum buffer size for the Photon is between 512 bytes and 1K.

The most important thing on the Photon is that you must handle the case when client.write() returns -16. This is the buffer full, try again later error. You can just attempt to write that block on the next iteration of loop and it will work great. You can delay(1) or delay(10) as well, it works about the same.

In theory, client.write() could return a number of bytes less than the buffer you passed it. I handled it my code, though I’ve never seen it happen on either the Photon or Electron. You’d just call client.write() again with the partial buffer if you wanted to handle it.

Handling those things I’ve sent terabytes of data at a rate of nearly 1 Mbyte per second off a Photon with no errors.

3 Likes

Awesome sketch, thanks!
Although I see you’re using a TCPClient. Here, I have a server to which a client connects and receives data. Would this make a difference? As you said, the server is calling the private member (client)'s block write method…

It works basically the same with TCPServer, as long as the data is going out of the Photon. Handle the -16 the same way. I’m not sure if I published that test, I’ll look.

@rickkas7, thank you for your precious help. I successfuly downloaded a 100MB file from the Photon without any errors by implementing the -16 check. I’m seeing speeds around 500kB/s. Here is my code for further reference to anyone interested:

	#define ERR_BUFFER_FULL (-16)
void getFile(String file){
	auto bytesRead =0; auto totalBytes = 0;
	uint8_t buf[256]; auto fileSize = fh->fileSize(file);
	do{
		bytesRead = fh->readFile(file, buf, sizeof(buf));
		auto writtenBytes = 0; auto lengthToWrite = sizeof(buf);
		if(totalBytes + sizeof(buf) >=fileSize) lengthToWrite = fileSize-totalBytes; // account for very last iteration, do not print buffer twice
		do{
			writtenBytes = dserver->write(buf, lengthToWrite);
		}while (ERR_BUFFER_FULL == writtenBytes);
		totalBytes+= writtenBytes;
	}while( (bytesRead == sizeof(buf)) && bytesRead > 0); // continue while still reading chunks of sizeof(buffer)from card..
	dclient.stop();
	fh->flush();
}
1 Like

Excellent! I also wrote a new TCPServer example. Without the overhead of reading from the SD Card I’m getting about 900 Kbytes/sec, or about 3.5 seconds for a 3 MB transfer from the Photon to the test server, which is what I expected. Also, this example is multi-client. I tested it at 1, 2, and 3 simultaneous connections to the TCP server and it works amazingly well. Basically, linearly, so with 3 connections you get about 300 Kbytes/sec. per connection. The Photon could probably go even higher in number of incoming connections, though I’m not sure why you’d want to.

#include "Particle.h"

const int MAX_CLIENTS = 5;
const int LISTEN_PORT = 80;
const int LINE_BUF_SIZE = 1024;
const unsigned long INACTIVITY_TIMEOUT_MS = 30000;
const int BYTES_TO_SEND = 3 * 1024 * 1024; // 3 MB

enum ClientState { CLIENTSTATE_READING, CLIENTSTATE_WRITING };

class ClientConnection {
public:
	ClientConnection();
	virtual ~ClientConnection();

	void loop();
	bool accept();

protected:
	void clear();
	void readRequest();
	void fillBuffer();
	void writeData();

private:
	ClientState state;
	char lineBuf[LINE_BUF_SIZE];
	bool inUse;
	TCPClient client;
	int readOffset;
	int writeOffset;
	unsigned long lastUse;
	byte nextToSend;
	int bytesSent;
};


String localIP;
TCPServer server(LISTEN_PORT);
ClientConnection clients[MAX_CLIENTS];

void setup() {
	Serial.begin(9600);

	// From CLI, use something like:
	// particle get test5 localip
	// to get the IP address of the Photon (replace "test5" with your device name)
	localIP = WiFi.localIP(); // localIP must be a global variable
	Particle.variable("localip", localIP);
	Serial.printlnf("server=%s:%d", localIP.c_str(), LISTEN_PORT);

	server.begin();
}

void loop() {
	// Handle any existing connections
	for(int ii = 0; ii < MAX_CLIENTS; ii++) {
		clients[ii].loop();
	}

	// Accept a new one if there is one waiting (and we have a free client)
	for(int ii = 0; ii < MAX_CLIENTS; ii++) {
		if (clients[ii].accept()) {
			break;
		}
	}
}


ClientConnection::ClientConnection() : inUse(false) {
	clear();
}

ClientConnection::~ClientConnection() {
}

void ClientConnection::loop() {
	if (!inUse) {
		return;
	}

	if (client.connected()) {
		switch(state) {
		case CLIENTSTATE_READING:
			readRequest();
			break;

		case CLIENTSTATE_WRITING:
			writeData();
			break;
		}

		if (millis() - lastUse > INACTIVITY_TIMEOUT_MS) {
			Serial.println("inactivity timeout");
			client.stop();
			clear();
		}
	}
	else {
		Serial.println("client disconnected");
		client.stop();
		clear();
	}
}

bool ClientConnection::accept() {
	if (inUse) {
		return false;
	}

	client = server.available();
	if (client.connected()) {
		Serial.println("connection accepted");
		lastUse = millis();
		inUse = true;
	}
	return true;
}

void ClientConnection::clear() {
	state = CLIENTSTATE_READING;
	lastUse = 0;
	readOffset = 0;
	writeOffset = 0;
	inUse = false;
	nextToSend = 0;
	bytesSent = 0;
}

void ClientConnection::readRequest() {
	// Read more of the line

	int toRead = LINE_BUF_SIZE - readOffset;
	if (toRead == 0) {
		Serial.println("invalid request line too long");
		client.stop();
		clear();
		return;
	}

	// Note: client.read returns -1 if there is no data; there is no need to call available(),
	// which basically does the same check as the one inside read().

	int count = client.read((uint8_t *)&lineBuf[readOffset], toRead);
	if (count > 0) {
		// Check for end-of-line characters
		Serial.printlnf("read %d bytes", count);
		readOffset += count;

		for(int ii = 0; ii < readOffset; ii++) {
			if (lineBuf[ii] == '\n') {
				// Found an end of line character
				if ((ii == 0) || (ii >= 1 && lineBuf[ii - 1] == '\r')) {
					// End of header, now send out the data
					Serial.println("got request line, returning response");
					fillBuffer();
					state = CLIENTSTATE_WRITING;
					return;
				}
			}
		}
		if (readOffset == LINE_BUF_SIZE) {
			// Filled entire buffer without getting CRLF
			Serial.println("invalid request");
			client.stop();
			clear();
		}
	}
}

void ClientConnection::fillBuffer() {
	for(size_t ii = 0; ii < sizeof(lineBuf); ii++) {
		lineBuf[ii] = nextToSend;
		nextToSend += 3; // arbitrary, but designed to each buffer won't have the same data in it
	}
}

void ClientConnection::writeData() {
	int toWrite = BYTES_TO_SEND - bytesSent;
	if (toWrite <= 0) {
		Serial.println("write complete!");
		client.stop();
		clear();
		return;
	}

	if (toWrite > (LINE_BUF_SIZE - writeOffset)) {
		// Writing more than 1024 bytes at a time seems to destabilize things, causing more errors
		// than smaller writes
		toWrite = (LINE_BUF_SIZE - writeOffset);
	}

	int count = client.write((uint8_t *)&lineBuf[writeOffset], toWrite);
	if (count == -16) {
		// Buffer full
		return;
	}
	else
	if (count < 0) {
		Serial.printlnf("write error %d", count);
		client.stop();
		clear();
		return;
	}

	// Normal case
	writeOffset += count;
	if (writeOffset >= LINE_BUF_SIZE) {
		// Sent all of the bytes in lineBuf, fill it with new data
		writeOffset = 0;
		fillBuffer();
	}
	bytesSent += count;

	lastUse = millis();
}


3 Likes