TCP Client read eventually stops getting packets, send continues to function

Hi,

Were testing out the TCP client functionality and running into a problem once the packet rate gets increased with the read saying there is no data. I've tested this on a Boron and an Argon.

Setup:

  • TCP client on particle side
  • TCP server in python to echo packet back

Observations:

  • packet rate of 1Hz has no issues
  • packet rate of 4Hz will sometimes cause the read to stop working
  • packet rate of 10Hz will cause the read to stop working within 5s to 1 min
  • when the read stops working (available() returns 0), the send still functions correctly and the server gets the packets to echo back
  • when the read stops working (available() returns 0) and I shut down the server, the particle does not detect that the connection has closed and continues to send packets
  • while the read is working correctly (available() returns > 0) and I shut down the server, the particle does detect that the connection has closed

Tests

  • set mode to Automatic and Semi Automatic
  • disconnect from cloud
  • system thread disabled

Will attach the code below, any ideas?

Thanks

Particle code

/* 
 * Project myProject
 * Author: Your Name
 * Date: 
 * For comprehensive documentation and examples, please visit:
 * https://docs.particle.io/firmware/best-practices/firmware-template/
 */

// Include Particle Device OS APIs
#include "Particle.h"

// Let Device OS manage the connection to the Particle Cloud
SYSTEM_MODE(SEMI_AUTOMATIC);
#if Wiring_WiFi
  STARTUP(WiFi.selectAntenna(ANT_EXTERNAL));
#endif

// Run the application and system concurrently in separate threads
SYSTEM_THREAD(DISABLED);

// Show system, cloud connectivity, and application logs over USB
// View logs with CLI using 'particle serial monitor --follow'
SerialLogHandler logHandler(LOG_LEVEL_INFO);

byte server[] = { X, X, X, X };
TCPClient wifiClient;
const int wifiClientPort{7150};

void wifiSetup()
{
  Log.info("STARTING WIFI SETUP");

  Particle.disconnect();
  waitUntil(Particle.disconnected);
  WiFi.connect(WIFI_CONNECT_SKIP_LISTEN);
  waitUntil(WiFi.ready);

  Log.info("WIFI setup DONE");
}

void wifiTcpClient()
{
  Log.info("WIFI TCP CLIENT START");

  while(!wifiClient.connect(server, wifiClientPort))
  {
      Log.info("WIFI TCP CLIENT CONNECTION FAILED -- SLEEP 1s");
      delay(1000);
  }

  waitUntil(wifiClient.connected);
  Log.info("WIFI TCP CLIENT CONNECTED");
}

void wifiTcpRead()
{
  static int count{0};
  static u_int8_t buf[40];
  static bool packetRead{false};
  
  count = wifiClient.available();
  if(count >= 40)
  {
      count = wifiClient.read(buf, 40);
      if(count != 40)
      {
          Log.info("DID NOT READ 40!!!");
          while(1)
          {
              delay(10000);
          }
      }
  }

  count == 40 ? packetRead = true : packetRead = false;
  if(packetRead)
  {
      Log.info("WIFI PACKET READ = %s", buf);
      packetRead = false;
  }
}

void wifiTcpWrite()
{
  static int packetCount{0};
  static char sendBuf[40] = "WIFI SENDING PACKET NUM";
  static unsigned long lastPacketTime{0};
  static const unsigned long packetInterval{75};

  if(!wifiClient.connected())
  {
      Log.info("WIFI TCP CONNECTION DROPPED -- NO WRITE");
      while(1)
      {
          delay(10000);
      }
  }

  if(millis() - lastPacketTime >= packetInterval)
  {
      snprintf(sendBuf, sizeof(sendBuf), "WIFI SENDING PACKET NUM=%d", ++packetCount);
      wifiClient.write((byte*)sendBuf, 40);
      Log.info("WIFI SENT PACKET = %s", sendBuf);
      lastPacketTime = millis();
  }
}

// setup() runs once, when the device is first turned on
void setup() {
  wifiSetup();
  wifiTcpClient();
}

// loop() runs over and over again, as quickly as it can execute.
void loop() {
  wifiTcpRead();
  wifiTcpWrite();
  delay(1);
}

Server code

import socket
import sys

# Create a TCP/IP socket
wifiSock = socket.socket(socket.AF_INET, socket.SOCK_STREAM)

# Bind the socket to the port
wifi_server_address = ('X.X.X.X', 7150)
print ('WIFI starting up on %s port %s' % wifi_server_address)
wifiSock.bind(wifi_server_address)

# Listen for incoming connections
wifiSock.listen(1)

while True:
    # Wait for a connection
    print ('waiting for a WIFI connection')
    wifi_connection, wifi_client_address = wifiSock.accept()

    try:
        print ('WIFI connection from', wifi_client_address)

        # Receive the data in small chunks and retransmit it
        while True:
            wifi_data = wifi_connection.recv(40)
            print ('WIFI received "%s"' % wifi_data)

            if wifi_data:
                print ('WIFI sending data back to the WIFI client')
                ret = wifi_connection.send(wifi_data)
                if(ret != 40):
                    exit(-1)
                
            
    finally:
        # Clean up the connection
        connection.close()

Particle printout of failure:

0001390168 [app] INFO: WIFI SENT PACKET = WIFI SENDING PACKET NUM=65
0001390211 [app] INFO: WIFI PACKET READ = WIFI SENDING PACKET NUM=65
0001390244 [app] INFO: WIFI SENT PACKET = WIFI SENDING PACKET NUM=66
0001390288 [app] INFO: WIFI PACKET READ = WIFI SENDING PACKET NUM=66
0001390320 [app] INFO: WIFI SENT PACKET = WIFI SENDING PACKET NUM=67
0001390364 [app] INFO: WIFI PACKET READ = WIFI SENDING PACKET NUM=67
0001390396 [app] INFO: WIFI SENT PACKET = WIFI SENDING PACKET NUM=68
0001390437 [app] INFO: WIFI PACKET READ = WIFI SENDING PACKET NUM=68
0001390471 [app] INFO: WIFI SENT PACKET = WIFI SENDING PACKET NUM=69
0001390516 [app] INFO: WIFI PACKET READ = WIFI SENDING PACKET NUM=69
0001390546 [app] INFO: WIFI SENT PACKET = WIFI SENDING PACKET NUM=70
0001390588 [app] INFO: WIFI PACKET READ = WIFI SENDING PACKET NUM=70
0001390622 [app] INFO: WIFI SENT PACKET = WIFI SENDING PACKET NUM=71
0001390664 [app] INFO: WIFI PACKET READ = WIFI SENDING PACKET NUM=71
0001390697 [app] INFO: WIFI SENT PACKET = WIFI SENDING PACKET NUM=72
0001390744 [app] INFO: WIFI PACKET READ = WIFI SENDING PACKET NUM=72
0001390772 [app] INFO: WIFI SENT PACKET = WIFI SENDING PACKET NUM=73
0001390818 [app] INFO: WIFI PACKET READ = WIFI SENDING PACKET NUM=73
0001390848 [app] INFO: WIFI SENT PACKET = WIFI SENDING PACKET NUM=74
0001390889 [app] INFO: WIFI PACKET READ = WIFI SENDING PACKET NUM=74
0001390924 [app] INFO: WIFI SENT PACKET = WIFI SENDING PACKET NUM=75
0001390965 [app] INFO: WIFI PACKET READ = WIFI SENDING PACKET NUM=75
0001391001 [app] INFO: WIFI SENT PACKET = WIFI SENDING PACKET NUM=76
0001391042 [app] INFO: WIFI PACKET READ = WIFI SENDING PACKET NUM=76
0001391076 [app] INFO: WIFI SENT PACKET = WIFI SENDING PACKET NUM=77
0001391119 [app] INFO: WIFI PACKET READ = WIFI SENDING PACKET NUM=77
0001391152 [app] INFO: WIFI SENT PACKET = WIFI SENDING PACKET NUM=78
0001391195 [app] INFO: WIFI PACKET READ = WIFI SENDING PACKET NUM=78
0001391227 [app] INFO: WIFI SENT PACKET = WIFI SENDING PACKET NUM=79
0001391272 [app] INFO: WIFI PACKET READ = WIFI SENDING PACKET NUM=79
0001391303 [app] INFO: WIFI SENT PACKET = WIFI SENDING PACKET NUM=80
0001391343 [app] INFO: WIFI PACKET READ = WIFI SENDING PACKET NUM=80
0001391379 [app] INFO: WIFI SENT PACKET = WIFI SENDING PACKET NUM=81
0001391420 [app] INFO: WIFI PACKET READ = WIFI SENDING PACKET NUM=81
0001391455 [app] INFO: WIFI SENT PACKET = WIFI SENDING PACKET NUM=82
0001391497 [app] INFO: WIFI PACKET READ = WIFI SENDING PACKET NUM=82
0001391530 [app] INFO: WIFI SENT PACKET = WIFI SENDING PACKET NUM=83
0001391573 [app] INFO: WIFI PACKET READ = WIFI SENDING PACKET NUM=83
0001391605 [app] INFO: WIFI SENT PACKET = WIFI SENDING PACKET NUM=84
0001391646 [app] INFO: WIFI PACKET READ = WIFI SENDING PACKET NUM=84
0001391682 [app] INFO: WIFI SENT PACKET = WIFI SENDING PACKET NUM=85
0001391725 [app] INFO: WIFI PACKET READ = WIFI SENDING PACKET NUM=85
0001391758 [app] INFO: WIFI SENT PACKET = WIFI SENDING PACKET NUM=86
0001391806 [app] INFO: WIFI PACKET READ = WIFI SENDING PACKET NUM=86
0001391833 [app] INFO: WIFI SENT PACKET = WIFI SENDING PACKET NUM=87
0001391873 [app] INFO: WIFI PACKET READ = WIFI SENDING PACKET NUM=87
0001391909 [app] INFO: WIFI SENT PACKET = WIFI SENDING PACKET NUM=88
0001391951 [app] INFO: WIFI PACKET READ = WIFI SENDING PACKET NUM=88
0001391985 [app] INFO: WIFI SENT PACKET = WIFI SENDING PACKET NUM=89
0001392030 [app] INFO: WIFI PACKET READ = WIFI SENDING PACKET NUM=89
0001392061 [app] INFO: WIFI SENT PACKET = WIFI SENDING PACKET NUM=90
0001392103 [app] INFO: WIFI PACKET READ = WIFI SENDING PACKET NUM=90
0001392137 [app] INFO: WIFI SENT PACKET = WIFI SENDING PACKET NUM=91
0001392178 [app] INFO: WIFI PACKET READ = WIFI SENDING PACKET NUM=91
0001392213 [app] INFO: WIFI SENT PACKET = WIFI SENDING PACKET NUM=92
0001392255 [app] INFO: WIFI PACKET READ = WIFI SENDING PACKET NUM=92
0001392288 [app] INFO: WIFI SENT PACKET = WIFI SENDING PACKET NUM=93
0001392328 [app] INFO: WIFI PACKET READ = WIFI SENDING PACKET NUM=93
0001392363 [app] INFO: WIFI SENT PACKET = WIFI SENDING PACKET NUM=94
0001392412 [app] INFO: WIFI PACKET READ = WIFI SENDING PACKET NUM=94
0001392438 [app] INFO: WIFI SENT PACKET = WIFI SENDING PACKET NUM=95
0001392485 [app] INFO: WIFI PACKET READ = WIFI SENDING PACKET NUM=95
0001392515 [app] INFO: WIFI SENT PACKET = WIFI SENDING PACKET NUM=96
0001392558 [app] INFO: WIFI PACKET READ = WIFI SENDING PACKET NUM=96
0001392590 [app] INFO: WIFI SENT PACKET = WIFI SENDING PACKET NUM=97
0001392635 [app] INFO: WIFI PACKET READ = WIFI SENDING PACKET NUM=97
0001392665 [app] INFO: WIFI SENT PACKET = WIFI SENDING PACKET NUM=98
0001392706 [app] INFO: WIFI PACKET READ = WIFI SENDING PACKET NUM=98
0001392740 [app] INFO: WIFI SENT PACKET = WIFI SENDING PACKET NUM=99
0001392784 [app] INFO: WIFI PACKET READ = WIFI SENDING PACKET NUM=99
0001392816 [app] INFO: WIFI SENT PACKET = WIFI SENDING PACKET NUM=100
0001392859 [app] INFO: WIFI PACKET READ = WIFI SENDING PACKET NUM=100
0001392892 [app] INFO: WIFI SENT PACKET = WIFI SENDING PACKET NUM=101
0001392934 [app] INFO: WIFI PACKET READ = WIFI SENDING PACKET NUM=101
0001392967 [app] INFO: WIFI SENT PACKET = WIFI SENDING PACKET NUM=102
0001393006 [app] INFO: WIFI PACKET READ = WIFI SENDING PACKET NUM=102
0001393042 [app] INFO: WIFI SENT PACKET = WIFI SENDING PACKET NUM=103
0001393089 [app] INFO: WIFI PACKET READ = WIFI SENDING PACKET NUM=103
0001393118 [app] INFO: WIFI SENT PACKET = WIFI SENDING PACKET NUM=104
0001393160 [app] INFO: WIFI PACKET READ = WIFI SENDING PACKET NUM=104
0001393194 [app] INFO: WIFI SENT PACKET = WIFI SENDING PACKET NUM=105
0001393236 [app] INFO: WIFI PACKET READ = WIFI SENDING PACKET NUM=105
0001393269 [app] INFO: WIFI SENT PACKET = WIFI SENDING PACKET NUM=106
0001393312 [app] INFO: WIFI PACKET READ = WIFI SENDING PACKET NUM=106
0001393344 [app] INFO: WIFI SENT PACKET = WIFI SENDING PACKET NUM=107
0001393393 [app] INFO: WIFI PACKET READ = WIFI SENDING PACKET NUM=107
0001393420 [app] INFO: WIFI SENT PACKET = WIFI SENDING PACKET NUM=108
0001393460 [app] INFO: WIFI PACKET READ = WIFI SENDING PACKET NUM=108
0001393495 [app] INFO: WIFI SENT PACKET = WIFI SENDING PACKET NUM=109
0001393537 [app] INFO: WIFI PACKET READ = WIFI SENDING PACKET NUM=109
0001393571 [app] INFO: WIFI SENT PACKET = WIFI SENDING PACKET NUM=110
0001393614 [app] INFO: WIFI PACKET READ = WIFI SENDING PACKET NUM=110
0001393647 [app] INFO: WIFI SENT PACKET = WIFI SENDING PACKET NUM=111
0001393690 [app] INFO: WIFI PACKET READ = WIFI SENDING PACKET NUM=111
0001393723 [app] INFO: WIFI SENT PACKET = WIFI SENDING PACKET NUM=112
0001393767 [app] INFO: WIFI PACKET READ = WIFI SENDING PACKET NUM=112
0001393799 [app] INFO: WIFI SENT PACKET = WIFI SENDING PACKET NUM=113
0001393841 [app] INFO: WIFI PACKET READ = WIFI SENDING PACKET NUM=113
0001393874 [app] INFO: WIFI SENT PACKET = WIFI SENDING PACKET NUM=114
0001393924 [app] INFO: WIFI PACKET READ = WIFI SENDING PACKET NUM=114
0001393950 [app] INFO: WIFI SENT PACKET = WIFI SENDING PACKET NUM=115
0001393992 [app] INFO: WIFI PACKET READ = WIFI SENDING PACKET NUM=115
0001394027 [app] INFO: WIFI SENT PACKET = WIFI SENDING PACKET NUM=116
0001394068 [app] INFO: WIFI PACKET READ = WIFI SENDING PACKET NUM=116
0001394102 [app] INFO: WIFI SENT PACKET = WIFI SENDING PACKET NUM=117
0001394152 [app] INFO: WIFI PACKET READ = WIFI SENDING PACKET NUM=117
0001394177 [app] INFO: WIFI SENT PACKET = WIFI SENDING PACKET NUM=118
0001394253 [app] INFO: WIFI SENT PACKET = WIFI SENDING PACKET NUM=119
0001394328 [app] INFO: WIFI SENT PACKET = WIFI SENDING PACKET NUM=120
0001394404 [app] INFO: WIFI SENT PACKET = WIFI SENDING PACKET NUM=121
0001394479 [app] INFO: WIFI SENT PACKET = WIFI SENDING PACKET NUM=122
0001394554 [app] INFO: WIFI SENT PACKET = WIFI SENDING PACKET NUM=123
0001394630 [app] INFO: WIFI SENT PACKET = WIFI SENDING PACKET NUM=124
0001394706 [app] INFO: WIFI SENT PACKET = WIFI SENDING PACKET NUM=125
0001394781 [app] INFO: WIFI SENT PACKET = WIFI SENDING PACKET NUM=126
0001394857 [app] INFO: WIFI SENT PACKET = WIFI SENDING PACKET NUM=127
^Cbash-5.1$ 

Server side still getting packets:

WIFI received "b'WIFI SENDING PACKET NUM=391\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00'"
WIFI sending data back to the WIFI client
WIFI received "b'WIFI SENDING PACKET NUM=392\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00'"
WIFI sending data back to the WIFI client
WIFI received "b'WIFI SENDING PACKET NUM=393\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00'"
WIFI sending data back to the WIFI client
WIFI received "b'WIFI SENDING PACKET NUM=394\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00'"
WIFI sending data back to the WIFI client
WIFI received "b'WIFI SENDING PACKET NUM=395\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00'"
WIFI sending data back to the WIFI client
WIFI received "b'WIFI SENDING PACKET NUM=396\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00'"
WIFI sending data back to the WIFI client
WIFI received "b'WIFI SENDING PACKET NUM=397\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00'"
WIFI sending data back to the WIFI client
WIFI received "b'WIFI SENDING PACKET NUM=398\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00'"

There is no guarantee that a 40-byte write in TCP will be received as a 40-byte read. It's stream protocol, and the boundaries are not preserved. Also you cannot simply wait until 40 bytes is available. I believe you should always read whatever is available and accumulate it into your own buffer to find the frame boundaries.

Also closing in TCPClient is confusing because the underlying TCP protocol is a two-step close that allows for a half-open connection (data flowing in one direction only). However the Arduino TCPClient implementation doesn't expose this and it can behave unpredictably. Also, because the FIN can be in the last data packet, if there's outstanding data to be read, you may not get the close notification until you've read it. The exact behavior is unpredictable, however, and sometimes you can get a close before you've read all of the data.

It's true that it's not guaranteed, but if anything came through I'd get 40 bytes eventually. In the case of 10hz, that never happens because the read buffer is empty.

As an update, we switched to a different platform and are able to run the above ported code at 100Hz with no issues. The connection issue is also resolved.

This topic was automatically closed 182 days after the last reply. New replies are no longer allowed.