Problems with Serial1 on an Electron

My company uses a custom radio module in our products, made by a 3rd party specifically for us. I have, as a proof of concept, connected this module to a photon on Serial1. The code then uses either Particle.publish or Serial.println to forward the data to a client. The code works perfectly and has been running for a couple of months in a soak test without any issues.

I’m now trying to use the same module with an electron so that we can deploy in remote settings. However I’m getting odd behaviour with the electron which would indicate that the radio module is either not receiving data correctly or that the electron is not receiving data from the module correctly, or both. The serial protocol requires ack/nack messages from the photon/electron and it could be that there’s a timing issue with the buffer on either side being corrupted. It’s hard to debug this but I wondered if anyone had any similar issues or had any insight? Our module requires 8N1 at 38400 baud.

Unfortunately the data that goes over the serial port is proprietary and I can’t share it in public - but would be happy to work offline with someone who could potentially help!

Just to fill in some info for whoever might be able to help.

Did you use SYSTEM_THREAD(ENABLED) in your Photon tests? Since this is not yet baked into the Electron factory firmware this might play a role.
Could you try any of the other UARTs on the Electron (Serial2, 4, 5)? Do you see the same behaviour?
What kind of odd behaviour do you see? Corrupted bytes, missed bytes, truncated transmissions, buffer artefacts, …?
Are you logging the raw input or only on successful transmissions?

@ScruffR I’m not using SYSTEM_THREAD(ENABLED). I’ve tried Serial4 and get the same behaviour. I’m expecting the receive buffer to fill with characters until a newline is received; then to send an acknowledgement message then wait until the next message starts to come in from the radio module. What’s actually happening is that 1: the radio module sometimes behaves as if it isn’t receiving the ACK messages and if it does receive it as if the message is corrupted. 2: sometimes I’m seeing the receive buffer on the electron contain garbage*, or to contain a partial message but followed by the ‘next’ message without the expected newline. (*By garbage I mean unexpected ascii characters)

As mentioned none of this is a problem on the photon.

OK, some things to work with :+1:

I’m sorry, some more questions:
How long are you comm lines?
Have you tried simple test sketch that only does the serial communication possibly only having the Electron talk to itself?
What length transmission do you expect? Are you getting anywhere near 64 byte?


Update:
When I try this sketch with my two Beta Electrons in a breadboard and a “self-talk” jumper wire (4x 20cm male/male -> breadboard -> male/male)and a the result seems fine.

char data[64];
char recv[128];

void setup() {
  Serial.begin(115200);
  Serial1.begin(38400);
  Particle.function("SerTest", test);
  Particle.variable("data", data);
  Particle.variable("received", recv);
}

void loop() {
  int i = 0;
  
  while(Serial1.available())
  {
    Serial.printf("%02x ", recv[i++] = Serial1.read());
  }
  if (i)
  {
    recv[i] = '\0';
    Serial.printf("\n<%s>\n", recv);
  }
}

int test(String cmd)
{
    cmd.toCharArray(data, sizeof(data));
 
    Serial.printf("\n<%s>\n", data); // leading and trailing newline for loggin 
    Serial1.printf("%s\n", data);    // trailing newline as end marker
    
    return cmd.length();
}

The Electron radio/modem talks over serial so I would bet that the serial interrupt handling timing is completely different between Photon and Electron.

Can you boil this down to test code that just uses loop-back on Serial1 with TX tied to RX on the Electron?

Then we can debug it and get @mdma or @BDub to help. Note that this is what @ScruffR suggested above too!

1 Like

This might be the reason why there is no Serial3 but only Serial1, Serial2, Serial4 and Serial5 ;-).

But as my (successful) test code above shows, there might be something else at play - I’d suspect some violation of the buffer integrity, which could well be down to a timing issue - possibly caused by the attention the modem needs after it lost contact to the towers.

@ScruffR the lines are about 5cm - is that what you meant? I have the module wires into a breadboard and swap in and out the photon and electron - so I don’t think it’s a physical issue. Each transmission is up to 46 bytes max. Sorry I don’t understand your ‘self-talk’ jumper wire - could you explain what I need to do again - I could understand needing 2 wires but not 4? Do I need 2 electrons?

Regarding timing, at the moment I’m only writing to Serial.println() for debugging - I’m not using Particle.publish() - so there shouldn’t be any use of the modem. Also I’m using serialEvent1() to get the incoming data - I’ll try while(Serial1.available()) tomorrow to see if that is any different. Could there be timing issues with writing to Serial as well as Serial1 - in the past I’ve tried using SoftwareSerial on an Arduino and had similar issues with garbled buffers.

I only used the four jumper wires to produce some "ill scenario" with a long fire with several joints adding further degradation of signal.
For a minimum test like my sketch, you'd only need one jumper wire that bridges RX and TX on the same device (hence self-talk).

But if you have a Photon on one side and an Electron on the other side you could go Electron TX -> Photon RX -> Phtoton TX -> Electron RX and back in the sketch to, to account to test if there is a slight difference in timing.

Normally the different serial ports don't interfere with eachother.

The thing with using serialEvent1 is, that it is not really an async event, as the name would suggest. It in fact only runs between iterations of loop(), so if you have a longish running loop() while RX data gets pushed in, your buffer might overflow if it doesn't get read at least once every 63 received bytes.

1 Like

Thanks for clarifying @ScruffR. I’ve used a single jumper wire and your sketch works for me too. Previously I didn’t have a loop function, just serialEvent1 which read the serial buffer until a linefeed was received and then called a function to process the buffer. I’ve removed that now and have a loop function as follows:

int reci = 0;
void loop()
{
    char c = '\0';
    while(Serial1.available())
    {
        c = Serial1.read();
        rcvbuf[reci] = c;
        reci++;
    }
    if(c == MESSAGE_END)
    {
        processBuffer(reci - 1);
        memset(rcvbuf, '\0', sizeof(rcvbuf));
        reci = 0;
    }
}

This again works on the Photon but not with the Electron. However, I’m not seeing corruption of the messages - just that the radio module doesn’t respond correctly and keeps re-initialising. I’ve started a dialogue with one of the engineers who’s responsible for the module to see what debugging I can get from that side. He’s in Sweden, I’m in the UK so the response may not be timely but I’ll keep you updated.

1 Like

@ScruffR we’ve worked with our supplier who’s suggested monitoring the serial traffic with an oscilloscope. What we’re seeing is that the electron’s Tx pin is being driven at over 4V and it’s likely that this is causing an issue as our module runs at 3v3. The Electron documentation asserts that the UART voltage is 3v3 so what are we missing - is there a software or hardware configuration that’s needed to peg the voltage at 3v3? We’re powering the module from the Electron’s 3v3 pin - there’s no other power involved - and the Electron’s Rx pin is reading correctly at 3v3.

I guess this is something for @BDub as he is the Particle shepherd for electrons on Electrons (aka Hardware Guy :wink: ).

How long are your communication lines?
Could it be that you are seeing some “resonance” between inductance of the wires and the capacitance of your system?
The output voltage of the Electron should not exceede 3.3V, but inductive pick-up (like an aerial) of your data lines and the LCR of your system might play a role here.
Common ground might also be something to check.

1 Like

@graemethc if you can upload a screenshot (even a cellphone pic) of what the TX line looks like on the oscilloscope with the Electron in your system? That would be a good starting point. Does it do the same thing without the Electron attached to your system?

1 Like

@BDub firstly this is the physical connection - as you can see the lines are pretty short:

Secondly this is what we’re seeing on the scope:

The top trace is the Tx line, the lower trace is the Rx line. As you can see the Rx line is around the correct voltage but the Tx line is 4V or so.

Ok one thing I’m noticing is it looks like your scope probes are not calibrated. The peak, reverse capacitive discharge curve should be more of a squarewave. To calibrate, hook the probes up to the reference waveform test point on the scope, and tune the capacitor built into each probe with a non-conductive screwdriver until it’s a flat squarewave.

It looks like your A signal is actually about 3.3 but the CH1 which is RX is 2.8V. Possibly because whatever is generating that signal has a bit lower of an output.

4 Likes

Thanks @bdub, @ScruffR: that’s the problem when you let software-heads loose on an oscilloscope! We calibrated the probes and yes, ahem, the voltages look good!

We compared traces on both Tx and Rx with a working Photon and can’t see much difference, though our scope isn’t a very high resolution one so we can’t be sure that we’re not missing something small. However I have set

SYSTEM_THREAD(ENABLED)

in the Electron code and this seems to have fixed the issue. Presumably there is some very marginal timing issue at work? The documentation suggests using a lot of caution with threading enabled - what should we be watching out for?

Currently on the Electron with v0.5.0-rc.1 I’ve seen some brief garbled debugging log output with threading enabled. I haven’t seen any issues with actual Serial1 or Serial characters via the Serial.printxxx() and Serial1.printxxx() though.

A good idea for debugging Serial stuff is to get an inexpensive Logic analyzer that has USART decoding capability (I would guess all of them do). Saleae Logic’s are nice which is what I use.

If you are sending too much data too fast, you can get overrun errors. You can try using blockOnOverrun(true) which is the default, for v0.5.0-rc.1 electron firmware. This was not available in v0.4.8-rc.6
https://docs.particle.io/reference/firmware/electron/#blockonoverrun-

This won’t however help if you are missing data when you are RX’ing from another source, such as a computer. The RX buffer is only 64 chars, so you have to keep up with emptying it.

You can also try a Software timer handler to do this job for you:

void process_serial()
{
    while (Serial1.available()) {
        // process Serial1 stuffs
    }
}

Timer timer(10, process_serial);

void setup()
{
    Serial1.begin(9600);
    timer.start();
}