TCPClient.print() vs. TCPClient.write()

Greeting Particlians :wink:

As mentioned before on this forum, Iā€™m working on a repository https://github.com/davidgatti/IoT-Raw-Sockets-Examples that explains in details how to take advantage of sockets using Particle and NodeJS.

I have a hard time understanding why there is a .print() and .write() method. I understand how they differ. But why not have one? Especially when the .print() one have issues handling a big buffer. Where .write() can handle buffers bigger then 2KB no problem.

print() has many more overloads than write(). Iā€™m not sure why there is a functional difference - print() delegates to write() to do the work.

Interesting. What to you mean by overloads?

Overload is an OOP term which is used when you have one function name, but with several different signatures (return types and/or parameter lists).

In a class you can have something like

classe myClass 
{
  ...
  void print(int val);
  void print(float val);
  void print(float val, int decimals);
  void print(const char* val);
}

all of those might do different stuff internally, but the compiler just chooses the one version of print() that fits the signature

e.g.

myClass x;

void loop()
{
  x.print(10); // this uses the (int val) version
  x.print(5.3, 2); // this uses the (float val, int decimals) version
}

But one major difference between print() and write() in your case might be that print(byte) might actually pick an overload that interprets byte as a number and hence ā€œtranslatesā€ it into the string representation (e.g. yourByte = 1; -> gets translated into print('1') which has the binary value 0b00110001 == 49 ) while write(byte) would always give you the original binary value 0b00000001

1 Like

Thank you for the Overload explanation. About the bits. Not sure this is the case. Iā€™ll write what I know, and understand, and will see if Iā€™m getting it wrong :slight_smile:

  1. you say that .write() will send data in binary, not sure, since the docs say send as a single byte (byte or char), where byte is just a unsigned char. My understanding is that this method sends just bytes converted to ASCII.
  2. .print() on the other hand will still sent everything as ASCII chars, but I can tell it to actually make it first binary. Meaning, if I send int 1, the .print() method will actually send char 00000001, where every digit is a single char.

And so, you say that .print() will call the .write() method in the end, which now explains why everything is sent as ASCII chars. But this means there is extra code that does something to the data before being passed to .write(). Because if I just .print() a buffer that is about 2KB, Iā€™ll get literally extra ASCII characters at random. For example

  • 31. Particle is a prototype-to-production platform for deveveloping an Internet of Things.: the word developing will get extra ev
  • 31. Particle is a prototype-to-production platform for devevelopi/ng an Internet of Things.: I get a /.

Any thoughts?

Iā€™m not seeing any difference between the print(const char *) and write(unsigned char *, size_t) versions when sending ASCII data.

Hereā€™s the test program I used. It sends 1024 byte buffers for ASCII characters from 32 to 126 (inclusive) to a server using TCPClient, and the server verifies that the bytes are received uncorrupted. I get the same over 900Kbytes/sec. for both print and write, and Iā€™m currently up to about 300 Mbytes sent on the connection and there have been no errors or corrupted bytes.

// Test app for sending large amounts of data over a TCP connection

#include "Particle.h"

// This test can be run with system thread enabled or disabled. It's a little faster
// with it enabled, about 1150 Kbytes/sec. vs. about 900 Kbytes/sec with it disabled
// (the default).
// SYSTEM_THREAD(ENABLED);

// Retained memory is used to keep track of reboots.
// PHOTON ONLY: Connect VBAT to GND to the retained memory will be initialized on power-up if
// you are not using a coin cell battery
// Do NOT connect VBAT to GND on an Electron or you will short out the 3.3V regulator!
STARTUP(System.enableFeature(FEATURE_RETAINED_MEMORY));

// Finite state machine states
enum State { STATE_CONNECT, STATE_FILL_BUFFER, STATE_SEND, STATE_RETRY_WAIT };

// Keeps track of the number of times we've run setup(), which should be equal to the number
// of reboots since we don't use sleep modes in this test app
retained int setupCount = 0;

// Various constants

// bufSize is the number of bytes we make in a typical write call. Making this 2048 or
// larger can cause data corruption on the Photon. 1024 seems experimentally to be ideal;
// if you make it smaller it works without errors but the data rate drops.
const size_t bufSize = 1024;

// Various timeout values.
const unsigned long retryWaitTimeMs = 2000;
const unsigned long sendTimeoutMs = 60000;

// Set to the IP address of the server to connect to
IPAddress serverAddr(192,168,2,4);
const int serverPort = 7123;

// Global variables
State state = STATE_CONNECT;
TCPClient client;
unsigned long stateTime = 0;
char buf[bufSize + 1];
size_t sentInBuf;
unsigned long totalSent = 0;
unsigned char bufStartChar;

void setup() {
	Serial.begin(9600);

	// Increment retained variable; should happen on every reboot (since we don't use sleep in
	// this program).
	setupCount++;
}

void loop() {
	switch(state) {
	case STATE_CONNECT:
		Serial.printlnf("** starting setupCount=%d millis=%lu", setupCount, millis());

		if (!client.connect(serverAddr, serverPort)) {
			// Connection failed
			stateTime = millis();
			state = STATE_RETRY_WAIT;
			break;
		}
		totalSent = 0;
		bufStartChar = (unsigned char) (32 + (setupCount % 95));
		state = STATE_FILL_BUFFER;
		// Fall through

	case STATE_FILL_BUFFER:
		// Each buffer has different bytes so we can detect corruption on the server
		for(size_t ii = 0; ii < bufSize; ii++) {
			buf[ii] = bufStartChar++;
			if (bufStartChar >= 127) {
				bufStartChar = 32;
			}
		}
		buf[bufSize] = 0;
		sentInBuf = 0;

		state = STATE_SEND;
		stateTime = millis();
		// Fall through

	case STATE_SEND:
		if (client.connected()) {
			// int count = client.write(&buf[sentInBuf], bufSize - sentInBuf);
			int count = client.print(&buf[sentInBuf]);
			if (count == -16) {
				// Special case: Internal buffer is full, just retry at the same offset next time
				// I'm pretty sure the result code for this is different on the Core, and probably the Electron.

				if (millis() - stateTime > sendTimeoutMs) {
					// I never hit this debug statement in my tests
					Serial.printlnf("** timeout sending sentInBuf=%u totalSent=%lu millis=%lu", sentInBuf, totalSent, millis());
					client.stop();
					stateTime = millis();
					state = STATE_RETRY_WAIT;
				}
			}
			else
			if (count > 0) {
				// Normal case, sent some bytes. count may be less than the buffer size, which
				// means we need to send the rest later.
				if ((size_t)count < (bufSize - sentInBuf)) {
					// This never seems to happen. You either get all of the bytes, or none of the bytes (-16).
					Serial.println("Partial send");
				}

				stateTime = millis();
				totalSent += count;
				sentInBuf += count;

				if (sentInBuf >= bufSize) {
					// Sent whole buffer, refill next time
					state = STATE_FILL_BUFFER;
				}
			}
			else {
				// Error
				Serial.printlnf("** error sending error=%d sentInBuf=%u totalSent=%lu millis=%lu", count, sentInBuf, totalSent, millis());
				client.stop();
				stateTime = millis();
				state = STATE_RETRY_WAIT;
			}
		}
		else {
			Serial.printlnf("** connection closed totalSent=%lu millis=%lu", totalSent, millis());
			client.stop();
			stateTime = millis();
			state = STATE_RETRY_WAIT;
		}
		break;

	case STATE_RETRY_WAIT:
		if (millis() - stateTime > retryWaitTimeMs) {
			// Wait a few seconds before retrying
			state = STATE_CONNECT;
			break;
		}
		break;
	}
}

I might have oversimplified the actual overloads in order to convay the idea rather than sticking strict to the actually present overloads.
As @rickkas7 correctly stated, there is a print(const char*) overload that does just the same as the respective write(), but when you just see something like

  for(int i=0; i < n; i++)
    Serial.print(x[i]);

You might assume a "non-translating" overload is picked by the compiler to send the byte data, but depending on the type of x[] you might be wrong (e.g. int x[10] will actually send the ASCII representation of each element's numeric value).
So above was mainly meant as a point to be aware of and keep in mind when using print() while write() rids you of that, since it only deals with bytes and might give you an error message rather than letting you believe what you think but do something different.

Everything is sent binary!
The convention that different collections of bits are sometimes seen as numeric values while the same bits can be viewed as text in other circumstances is something owed to interaction with humans who need a "readable" representations in order to comprehend the data produced inside the binary machine.
We humans need to translate a binary value of 0b00010001 to another binary representation of 0b00110001 00110111 (this is what print() sometimes does, depending on chosen overload) which will be sent over a serial interface to be printed on screen as "17", while a machine will be happy to accept the original binary pattern (this is what some print() overloads do, but write() always does) and just do its job with it.

After having said all this, the sample of scrambled data you've shown above would rather suggest a sporadic issue with the transmission than a systematic problem revolving around the difference between print() and write().
It might have to do with the speed you are writing and reading the data, the size of the buffers involved, the quality of transmission, ...

If this is the case, then why this happens only with .print() and never with .write()? while sending the exact same data on a local network with 3 devices connected to the router.

Everything that you wrote is obvious but also you write that the compiler might choose a different method, not true, since I can explicitly say in the second parameter of the .print() method how the data should be interpreted.

Anyway, I'll try later to rewrite my example and paste it here.

No, this is a misconception.
Please don't mix two things here and don't imply that your understanding of the matter (based on interpretation of the result) outweighs the factual understanding of OOP, C++, compilers and the framework source code.

What second parameter are you refering to? Is it the on that can take BIN, DEC or HEX?
If so, what you are refering to is after the compiler has already decided to use the two parameter overload of the function. You then can only provide that kind of parameter to it and exactly that same chosen overload can interpret the value of the parameter. If you had this parameter be a variable (rather than a literal) then the compiler had no way to know what "overload" to choose, since this is a compile time decision, but the parameter has only a run-time value.

But if everything I wrote is that obvious, so be it.

1 Like

@ScruffR I see that the word obvious was interpreted as something negative. I didnā€™t mean for my answer to be negative. My goal was to say, that I understand the binary conversion that you explained, and the idea was to let you know that I get that.

Sorry for the misunderstanding. Language is hard :slight_smile: and the internets make it harder :wink:

If youā€™ll accept my apologies I would like to keep the conversation going, and do a followup with a code example that causes the issue for me, and my hope is that an example will help show what is happening on my side.

1 Like

OK, both communicating in their non-native language might explain things :sunglasses:

So back to the topic:
It would be good to see a short code sample that demonstrates the issue.
Meanwhile you could have a look at the different implementations of the multiple print() overloads used by TCPClient and other Stream based classes.
https://github.com/spark/firmware/blob/develop/wiring/inc/spark_wiring_print.h
https://github.com/spark/firmware/blob/develop/wiring/src/spark_wiring_print.cpp

Youā€™ll notice (in spark_wiring_print.h) that write() is marked virtual while the print() overloads are not.
This is due to the fact, that write() will most likely be reimplemented in the derived classes since each class might have different needs, while print() performs more ā€œabstractā€ (e.g. non-hardware related) tasks and only at the end internally calls the class specific write(). While write() just takes the raw bytes and forwards them without much ado.

So if you see issues with print() and not with write() itā€™s most likely due to these ā€œabstractā€ actions performed before handing the data over to write().

Here are some more links to dive into the implementation
https://github.com/spark/firmware/blob/develop/wiring/src/spark_wiring_tcpclient.cpp
https://github.com/spark/firmware/blob/develop/hal/src/photon/socket_hal.cpp

sock_result_t socket_send(sock_handle_t sd, const void* buffer, socklen_t len)
{
    sock_result_t result = SOCKET_INVALID;
    socket_t* socket = from_handle(sd);
    if (is_open(socket)) {
        wiced_result_t wiced_result = WICED_TCPIP_INVALID_SOCKET;
        if (is_tcp(socket)) {
            wiced_result = wiced_tcp_send_buffer(tcp(socket), buffer, uint16_t(len));
        }
        else if (is_client(socket)) {
            tcp_server_client_t* server_client = client(socket);
            wiced_result = server_client->write(buffer, len);
        }
        if (!wiced_result)
            DEBUG("Write %d bytes to socket %d result=%d", (int)len, (int)sd, wiced_result);
        result = wiced_result ? as_sock_result(wiced_result) : len;
    }
    return result;
}

All of the above might not be explained technically exact with the correct terminology, but the general idea should be clearer that way.

@ScruffR @mdma @rickkas7 please accept my apologies, I did run the code again, and now it just works. I have no idea what whenā€™t wrong before :flushed:.

@ScruffR thank you also for the last post.

This is the code if someone is interested

TCPClient client;

int port = 1337;
byte server[] = { 192, 168, 1, 100 };

char* txt = "1. Particle is a prototype-to-production platform for developing an Internet of Things\n2. Particle is a prototype-to-production platform for developing an Internet of Things\n3. Particle is a prototype-to-production platform for developing an Internet of Things\n4. Particle is a prototype-to-production platform for developing an Internet of Things\n5. Particle is a prototype-to-production platform for developing an Internet of Things\n6. Particle is a prototype-to-production platform for developing an Internet of Things\n7. Particle is a prototype-to-production platform for developing an Internet of Things\n8. Particle is a prototype-to-production platform for developing an Internet of Things\n9. Particle is a prototype-to-production platform for developing an Internet of Things\n10. Particle is a prototype-to-production platform for developing an Internet of Things\n11. Particle is a prototype-to-production platform for developing an Internet of Things\n12. Particle is a prototype-to-production platform for developing an Internet of Things.\n13. Particle is a prototype-to-production platform for developing an Internet of Things.\n14. Particle is a prototype-to-production platform for developing an Internet of Things.\n15. Particle is a prototype-to-production platform for developing an Internet of Things.\n16. Particle is a prototype-to-production platform for developing an Internet of Things.\n17. Particle is a prototype-to-production platform for developing an Internet of Things.\n18. Particle is a prototype-to-production platform for developing an Internet of Things.\n19. Particle is a prototype-to-production platform for developing an Internet of Things.\n20. Particle is a prototype-to-production platform for developing an Internet of Things.\n21. Particle is a prototype-to-production platform for developing an Internet of Things.\n22. Particle is a prototype-to-production platform for developing an Internet of Things.\n23. Particle is a prototype-to-production platform for developing an Internet of Things.\n24. Particle is a prototype-to-production platform for developing an Internet of Things.\n25. Particle is a prototype-to-production platform for developing an Internet of Things.\n26. Particle is a prototype-to-production platform for developing an Internet of Things.\n27. Particle is a prototype-to-production platform for developing an Internet of Things.\n28. Particle is a prototype-to-production platform for developing an Internet of Things.\n29. Particle is a prototype-to-production platform for developing an Internet of Things.\n30. Particle is a prototype-to-production platform for developing an Internet of Things.\n31. Particle is a prototype-to-production platform for developing an Internet of Things.\n32. Particle is a prototype-to-production platform for developing an Internet of Things.\n33. Particle is a prototype-to-production platform for developing an Internet of Things.\n34. Particle is a prototype-to-production platform for developing an Internet of Things.\n35. Particle is a prototype-to-production platform for developing an Internet of Things.\n36. Particle is a prototype-to-production platform for developing an Internet of Things.\n37. Particle is a prototype-to-production platform for developing an Internet of Things.\n38. Particle is a prototype-to-production platform for developing an Internet of Things.\n39. Particle is a prototype-to-production platform for developing an Internet of Things.\n40. Particle is a prototype-to-production platform for developing an Internet of Things.\n41. Particle is a prototype-to-production platform for developing an Internet of Things.\n42. Particle is a prototype-to-production platform for developing an Internet of Things.\n43. Particle is a prototype-to-production platform for developing an Internet of Things.\n44. Particle is a prototype-to-production platform for developing an Internet of Things.";

void setup() {

    Serial.begin(9600);

    Serial.println("UP");

    // Connect to the remote server
    if(client.connect(server, port)) {

        Serial.println("Connected");


    } else {

        // if we can't connect, then we display an error.
        Serial.println("error");

    }

}

void loop() {

    // Send our data to the remote server
    client.print(txt);

    // Send the separator
    client.print(',');

    Serial.println("Done");

    delay(100);

}

I have one last question :slight_smile: why does the print method exist? Is it just to make it easier to send data?

That's one point but the bigger one is that this exists in Wiring and hence on Arduinio and hence also on platforms that keep close to that framework.
And once you know how things work together you'd not want to miss it anymore.

1 Like

Got it :smile: :bow: