Serial1 and udp receive interference on argon with ethernet?

hello,

we have an argon product which speaks modbus (over Serial1) and ethernet (with the Adafruit ethernet board)

we talk to a modbus device every second and listen for udp messeges. this works fine as long as there is not much udp traffic to the device (udp broadcasts from another device)

increasing the udp traffic will result in modbus errors. the buffers will be mangled up. we reduced the modbus request to the bare minimum (static serial writes and reads)

has anyone seen this too? any fixes or advice?

thanks
frank

Hi Frank,

Could you please provide the error(s) you’re encountering, just so we could maybe better answer your question.

Thank you,
Lance

hi lance

with the modbus library we see different errors (invalid slave, crc, timeout). to debug this we write a single modbus request as serial writes (representing a modbus request) followed by reads: we see the reads having missing parts/bytes or having extra parts/bytes when udp traffic increases

most of the modbus request go through fine. the modbus device we address is something we have used for a ling time and never gave us trouble with the same request sequences. without udp traffic it all works fine

any ideas?

thanks
frank

best situation we got so far is reading all udp packages just before doing the serial/modbus operation. in this case we see about 1% errors in de modbus request (versus virtual none in case of no udp)

I’d guess the UDP packets are delaying the loop long enough to overflow the serial buffer, which is only 64 bytes. What I would do is run only the modbus serial decoding in a worker thread so the other system operations would not affect getting the data from serial at a constant rate.

the modbus response (to the request that we are testing this with) is 21 bytes. we only do this once a second. we should never hit the 64 bytes…

also we not only lose bytes: sometimes we get extra bytes or data is scrambled

modbus/Serial1 baud rate is 9600. we use udp broadcast to communicate between the nodes

this is how we read the modbus response (just for this test case):

        // 5 bytes, 1+1.5 ms each = 12.5ms
        Serial1.setTimeout(15);
        bytesRead = Serial1.readBytes(rxBuffer,5);

        // 21-5 bytes, 1+1.5 ms each = 40ms
        Serial1.setTimeout(45);
        bytesRead = Serial1.readBytes(rxBuffer,21-5);

Hi,
ok so far we know that you have some issue with your modbus device response

  1. how do you handle the request ? I mean the full request is coming through UDP? (build at the host side ?) e.g : 12 03 4001 002 or you have defined, for example an array of requests and UDP just points which one should be used
  2. how are you checking for CRC ?
  3. which registers you are trying to read/set ?
  4. if requests are build on the host side are you able to confirm/check if they are always correct ? not corrupted ?

as more info we can get about your code, setup etc. as much we can try to help.
Right now is just a guessing game

1 - there is modbus over serial1 and there is udp. they have a different function but do seem to interfer. all we do now on the argon is a udp read of we see a packet. no processing. the modbus request is just a fixed serial write sequence then change the direction pin and the reads as in the previous post. no udp no problem

2 - the actual modbus code does have crc checken. right now in our test case (with fixed request) we just check the data coming in as to see exactly where the problem is

3 - this is a modbus fixed request to a device that we have been using for a long time (without udp) without any problems. same kind of reads every second for days without any error

4 - the problem is that the error rate is much higher than without udp but not so high i can put a scope on the signal and check. but it seems highly unlikely that the outgoing request or the addressed device is the problem as we see data coming in that looks like the response just with data added, changed or missing

it is a guessing game for us also… the test code is very simple. the actual part that seems to fail is the two reads already posted. the udp parts is really nothing, just getting the packets

What about RF interference? Obviously RS485 is pretty tolerant, so I’m more thinking about the lines between the modus transceiver and the UART port, and also the SPI lines to the Wiznet chip. This is completely grasping at straws, but maybe try changing the SPI data rate and see if has an effect? I don’t really think that is it, but if you’re out of ideas…

hi rickkas, will try. this might actually be something we can see on the scope. thanks!

Hi, so I guess I got you the UDP is not involved with modbus at all that’s corect? You have a fixed request for a modbus device and UDP is doing something on side ?
So 2 tings which is in my mind:

  1. You are using the Stream Class with your serial1 but what I read on reference is all are relay on that class I mean serial and UDP I’m so weak regarding to the Classes :man_facepalming:t3: Just know that they are some high abstract level of memory organization/representation
    Maybe there’s something regarding to this :thinking:
  2. Another thing is about RF which were pointed by @rickkas7
    So what about if you gona synchronize all tings ? I mean you need 60ms for a reading back the answer and some ms for send request let’s say another 60ms
    Total is 120ms and you stil have 880ms where UDP can act and in this time you can send as much as possible UDP packets then shut down all SPI and UDP stuff do Serial1 things and again. This test can show us if it’s related to RF interference :man_shrugging:t3:

correct, modbus is over Serial1 (and RS485 driver) and UDP is over ethernet using the Adafruit ethernet featherwing – functionally they have nothing to do with eachother…

1 - in the code excerpt above we use the stream class. but using the single byte write/read/available gives about the same results

2 - we are going to check electrical noise on the modbus lines. as for your idea about synchronizing, this will be difficult: we cannot schedule when UDP packets arrive. i guess we can test this by not allowing any interaction with the ethernet chip. will check this. i read something about how to do this.

thanks thinking about this!
cheers
frank

So one thing to note here (at least for myself for future reference) - I’ve been spending the past few days trying to debug issues with serial communication (among other things), particularly when attempting to start setting up serial communication in its own thread.
One thing to note, is that when I set the serial worker thread to the default priority, I would encounter situations where the first few bytes of data were either not captured correctly, or were corrupted - despite the serial monitor hooked up showing no issues with data on the bus - data was being sent from the argon to target(s) correctly, and from target(s) to argon correctly.
Setting increasing the priority slightly over the default (literally +1 higher) allows the serial to capture data without any issues.
Considering that the worker thread is the only consumer/user of Serial1, and there is no indication of buffer overruns (and the receipt of data does work, some of the time), it’s a bit of a mystery to me why this is occurring - and why stepping up the priority would be a solution.
If I get the opportunity, I’ll see what I can do about doing a minimum viable threaded version, and see if it encountered the same issues, but it is odd - particularly since I generally believe that changing the priority is not a solution to an underlying problem- it probably means I’m missing some interaction.
Ah well.

What version of Device OS are you using? There’s a bug fixed in 4.0.0, 3.0.0, and 2.1.0 that might affect UART reads and threading. However, it very well may be something completely different.

I’m using 4.0.0-4.0.2 - and I’m observing an absolutely night/day difference in ability to communicate. I’ll see if I can do a simple example where we can see an A/B difference

OK: here’s some code and screenshots of results - note, you’ll need to adjust pin values based on your setup/configuration.

Summary
/*
 * Project SerialCheckout
 * Description: Demonstrates observed error on reading serial from thread with ethernet enabled unless priority of thread is increased by 1
 * Author: JGU
 * Date: 27 March 2023
 */
SYSTEM_MODE(SEMI_AUTOMATIC);
SYSTEM_THREAD(ENABLED);
PRODUCT_VERSION(1);
#define ERRMSG_LEN_128B_MAX 128
#define SERIAL_ID_LENGTH_PARTICLE 64
#define SER_BUFF_LEN_DEMO 256
#define SER_BUFF_STR_DEMO 1024
#define MAX_TLG_LEN 20

#define RS485_DIR_PIN D2
#define RS485TX true
#define RS485RX !RS485TX

#define STX 0x02
#define ETX 0x03
SerialLogHandler loghandler;
Logger mainLogger("app.main");

#define THREADED_SERIAL_DEMO
//#define SHOW_ME_THE_ERROR

#ifdef THREADED_SERIAL_DEMO
void thread_f();
#endif 


// setup() runs once, when the device is first turned on.
void setup() {
  // Put initialization like pinMode and begin functions here.
  System.enableFeature(FEATURE_ETHERNET_DETECTION);
  System.enableFeature(FEATURE_RESET_INFO);

  Particle.keepAlive(20); // attempting a lower keepalive value to get rid of online/offline spam
  /* Attempting to force MTU to lower setting on ETH - this is normally 1500, but that can be problematic on celluar systems */
  if_t iface = nullptr;
  if_get_by_index(Ethernet, &iface);
  if_set_mtu(iface, 1379);
  /* end MTU adjust */
  Serial.begin();
  pinMode(D8,INPUT);
  pinMode(D7,INPUT);
  pinMode(D6,INPUT);
  // Configure pin direction and level
	pinMode(RS485_DIR_PIN, OUTPUT);
	digitalWrite(RS485_DIR_PIN,LOW); //receive is LOW, transmit is HIGH - set to LOW by default(RECEIVE) at start
	Serial1.begin(115200, SERIAL_8N1);


  
  waitFor(Serial.isConnected, 1000);
  int rstReason = System.resetReason();
  char errmsg[ERRMSG_LEN_128B_MAX];

  char tmp[SERIAL_ID_LENGTH_PARTICLE] = {};
  int sz = hal_get_device_serial_number(tmp, sizeof(tmp), nullptr);
  if (sz == 0) {
    strcpy(tmp,"BadSerial");
  } 
  Particle.connect();

  switch(rstReason){
    case RESET_REASON_UPDATE:
    case RESET_REASON_DFU_MODE:
    case RESET_REASON_USER:
    case RESET_REASON_POWER_DOWN:
    case RESET_REASON_PIN_RESET:
      //these are of no particular concern, and part of a normal operation
      sprintf(errmsg,"-- Reset Reason was considered a normal situation -- :%d",rstReason);
      mainLogger.info(errmsg);
      break;
    default:
      sprintf(errmsg,"RSTFAIL - bad reason for reset! %d",rstReason);
      mainLogger.error(errmsg);
      break;
  }

  #ifdef THREADED_SERIAL_DEMO
  #ifdef SHOW_ME_THE_ERROR
  new Thread("armCommThd", thread_f);
  #else
  new Thread("armCommThd", thread_f,OS_THREAD_PRIORITY_DEFAULT+1,OS_THREAD_STACK_SIZE_DEFAULT_NETWORK);
  #endif
  #endif


}

// loop() runs over and over again, as quickly as it can execute.
void loop() {

  // The core of your code will likely live here.
  static uint32_t lastMillis = 0;
  static uint32_t longMillis =0;
  uint32_t curMillis=millis();   

  #ifndef THREADED_SERIAL_DEMO
  static unsigned char receivedData[SER_BUFF_LEN_DEMO] = {0};
  static uint8_t curpos = 0;
  while(Serial1.available()){
      receivedData[curpos] = (unsigned char)Serial1.read();
      curpos+=1;
  }

  if (curMillis - lastMillis  > 30)
  {
    lastMillis=curMillis;
    char infoMsg[SER_BUFF_STR_DEMO] = {0x00};
    sprintf(infoMsg,"ReceivedData:");
    uint8_t readpos = 0;
    while (!(readpos == curpos)){
      sprintf(infoMsg+strlen(infoMsg),"%02X",receivedData[readpos]);
      readpos+=1;
    }
    mainLogger.info("%s",infoMsg);
    memset(infoMsg,0x00,SER_BUFF_STR_DEMO*sizeof(char));
    memset(receivedData,0x00,SER_BUFF_LEN_DEMO*sizeof(unsigned char));
    curpos = 0;
    uint8_t outmsg[MAX_TLG_LEN] = {0};
    uint8_t dCnt = 0;
    outmsg[dCnt++] =STX;
    outmsg[dCnt++] =0x36;
    outmsg[dCnt++] =0x10;
    outmsg[dCnt++] =0x04;
    outmsg[dCnt++] =0x01;
    outmsg[dCnt++] =0x00;
    outmsg[dCnt++] =0x95; 
    outmsg[dCnt++] =0x8c;
    outmsg[dCnt++] =ETX;

    digitalWrite(RS485_DIR_PIN,HIGH); //set the RS485 direction for enabling transmit
		delayMicroseconds(100); //THVD1410 has enable time in the order of 14us, so giving margin

    Serial1.write(outmsg, dCnt);
    Serial1.flush();
    digitalWrite(RS485_DIR_PIN,LOW);//set the RS485 direction for receive after writing is complete
  }
  #endif
  if (curMillis-longMillis> 2000){
    Particle.publish("test","bleh",NO_ACK); // forcing some form of activity out the ethernet
    longMillis=curMillis;
  }

}

#ifdef THREADED_SERIAL_DEMO
void thread_f(){
    static uint32_t lastMillis = 0;
    static uint32_t longMillis =0;

    
    static unsigned char receivedData[SER_BUFF_LEN_DEMO] = {0};
    static uint8_t curpos = 0;
    while(true){
      uint32_t curMillis=millis();   

      while(Serial1.available()){
        receivedData[curpos] = (unsigned char)Serial1.read();
        curpos+=1;
      }
      
      if (curMillis - lastMillis  > 30)
      {
        lastMillis=curMillis;
        char infoMsg[SER_BUFF_STR_DEMO] = {0x00};
        sprintf(infoMsg,"ReceivedData:");
        uint8_t readpos = 0;
        while (!(readpos == curpos)){
          sprintf(infoMsg+strlen(infoMsg),"%02X",receivedData[readpos]);
          readpos+=1;
        }
        mainLogger.info("%s",infoMsg);
        memset(infoMsg,0x00,SER_BUFF_STR_DEMO*sizeof(char));
        memset(receivedData,0x00,SER_BUFF_LEN_DEMO*sizeof(unsigned char));
        curpos = 0;
        uint8_t outmsg[MAX_TLG_LEN] = {0};
        uint8_t dCnt = 0;  // for the purposes of the demo we aren't checking for potential overwrite condition, so don't be stupid with how many bytes sent in a message
        outmsg[dCnt++] =STX;
        outmsg[dCnt++] =0x36;
        outmsg[dCnt++] =0x10;
        outmsg[dCnt++] =0x04;
        outmsg[dCnt++] =0x01;
        outmsg[dCnt++] =0x00;
        outmsg[dCnt++] =0x95; 
        outmsg[dCnt++] =0x8c;
        outmsg[dCnt++] =ETX;

        digitalWrite(RS485_DIR_PIN,HIGH); //set the RS485 direction for enabling transmit
        delayMicroseconds(100); //THVD1410 has enable time in the order of 14us, so giving margin

        Serial1.write(outmsg, dCnt);
        Serial1.flush();
        digitalWrite(RS485_DIR_PIN,LOW);//set the RS485 direction for receive after writing is complete
      }
      os_thread_yield();
    } 
}
#endif

Expected operation looks like this (this is how it looks in normal loop, or priority +1) (undefine SHOW_ME_THE_ERROR, and define/undefine THREADED_SERIAL_DEMO if you want it to run as a thread or in main application loop):

This is how it looks when the thread has normal default priority (define THREADED_SERIAL_DEMO and SHOW_ME_THE_ERROR):

Note how behavior of main loop and threaded with priority +1 have no issues, but threaded with priority default does.

I think you might be running into a bug in os_thread_yield() that causes it to not yield to a thread of equal priority, or something similar to that. You can test out this theory by using delay(1) instead of thread yield.

I’ll give that a shot the next time I have the setup in place, which will either be tomorrow or sometime next week.