Intermittent but recurrent Xenon mesh connectivity issues

[jmag]

I setup my recently arrived Mesh hardware, and although the Argon has worked very well the Xenon is causing all sorts of issues with connectivity.

All are running 0.8.0-rc.27 firmware.
Connecting to local WiFi through Argon reliably and also by setting Wifi Credentials in the firmware it can be taken from one place to another and connects happily and fast to the new networks.

Xenon - Android app used to enrol in mesh - slow, but fine. Started off running Tinker, as expected after the updates ran.

Then I ran the Photoresistor example code on the Particle Web IDE, which could see the device, but after each flash it would still be running Tinker, even though the LED sequence looked right.

Multiple resets later (button, pull power) it flashes and runs the code, but each time I go to update it is unreliable, showing the correct LED sequence, but still running the old code.

Checked firmware refresh through CLI, ran through the SSL key check procedure, reset.

Now when started it MAY connect and send a few publish events, but then goes into a cycle of connecting and disconnecting.

Startup - takes 3:55
Green flash
fast cyan flash
orange x3
slow cyan x 21
red x 1
fast cyan
slow cyan ~20
orange x 3
slow cyan ~20
orange x 3
slow cyan ~20
orange x 3
slow cyan ~ 20
orange x 3
slow cyan ~5
fast cyan
breathing

Then after about 10 minutes it does this, with a repeat each ~12:00
breathing
green x 2
fast cyan x 3
red x 1
v fast cyan ~2 sec
slow cyan ~20
orange x 3
slow cyan ~20
orange x 3
slow cyan ~4
fast cyan ~20sec
red x 1
slow cyan ~4
red x 1
slow cyan ~6
fast cyan ~20 sec
red x 1
1 x green
slow cyan x 4
fast cyan ~3 sec
red x 1
slow cyan 10sec
fast cyan 20sec
red x 1
green x 1
slow cyan ~10 sec
fast cyan ~5 sec
red x 1
slow cyan ~10 sec
fast cyan ~20 sec
breathing

All of which takes 5:02

Then it all repeats.

With the 3 x orange and the 1 x red error codes I tried the Red Blink Basic Errors procedure here: https://docs.particle.io/tutorials/device-os/led/xenon/, and it worked, without errors, but the same stuff happened afterwards.

This occurs with the units physically close together as well as 10-20m apart, on different power supplies, on battery or off.

Any suggestions would be gratefully accepted as I am very keen to use the mesh functionality.

[KyleG]

Hello,

Let me ping someone that might be able to help, @ParticleD, or @mstanley are you able to assist?

[mstanley]

Hey jmag,

That’s unusual behavior indeed. Do you have other Xenons as well experiencing this issue? Or is this your only Xenon?

Have you attempted to flash your user code to the Xenon using the CLI as well? Or only OTA through the web IDE?

I’d like you to attempt to try and flash it through the CLI as well. If this behavior persists even with CLI flashing, it may be an indication of device failure and I would recommend you file a support ticket referencing this topic.

Also, would you be able to provide the device Id? You may do so here or in private message with me directly.

[jmag]

Hi Matthew,
Thanks for your reply.

I have two others and will starting giving one a try... as well as trying the CLI for flashing.

I found this post which implies similar behaviour, but on an earlier firmware:

Xenon only Mesh Experiment with BUG and Caveat

I’ve also found that when the Xenons are not connected to the cloud they are far more stable with the current firmware. Publishing this way to an Argon using Mesh.publish and having the Argon convert to a Particle.publish results in a lot less flashing cyan with intermittent red blinks.

Sorry for my ignorance, but how to I PM you on this site?

Regards,
John.

[mstanley]

Hey jmag,

No worries!

For Discourse, you should be able to message another member on the site by clicking on their icon in the top left hand corner of the post. From there, a modal should pop up with a message option in a blue box on the right hand side.

[jmag]

Hi Matthew,
The behaviour persists even after flashing from the CLI.

I have yet to enroll a new Xenon and test it.

Regards,
John.

[jmag]

Hi Matthew,
I enrolled another Xenon - which took longer because of the instructions on the current app being out of date:
It is particularly confusing as the App (current version as of today) no longer has the message down the bottom that says “ASSISTING DEVICE BLINKING BLUE” [Ed - caps in original] as is shown in this Youtube video:

Unfortunately, it does the same thing WRT regular disconnects and variations on the same sequence as it tries to and eventually reconnects - this time after ~ 4 minutes. Its initial connect time and also the reconnect time is shorter but ~ 1/2 for what that is worth.

It was flashed from the CLI - I didn’t even try the Web IDE.

[mstanley]

Hey jmag,

I did some digging on this from the cloud side and do notice that the device ID of the Xenon you provided shows a lot of debug logging of IP changes (primarily the port) on the backend. As far as connection stability issues, I suspect it may be a router configuration issue.

If you can this on a different network or perhaps even attempt setting up your gateway Argon to tether off your phone, this would be useful information to see if the mesh network and nodes are behaving as intended.

Furthermore, there are some known connectivity and re connectivity issues identified with mesh that can be found in this Github issue at: https://github.com/particle-iot/device-os/pull/1661

It is possible some of the reliability issues are related to this issue. While merged, this issue has not been publicly released yet. My understanding is the ambition for this release (rc28, tentatively), is hopefully in the next few weeks.Hopefully it’s not too much to ask, but I would like to await the release of rc28 and have you update your devices to it and see if a change in behavior is noticed.

If the wait is an issue, do let me know and I can see what other arrangements can be made.

[jmag]

Hi Matthew,
I will try the units out on a new network, with different routers, etc tomorrow.

In the meantime I have done more testing and found that when the second Xenon is <2-3m away from the Argon, it connects almost immediately and starts publishing, reliably.

If I move it 5-7m away it begins the fast cyan / red / fast cyan cycle and eventually goes into the fast cyan / 3 orange / fast cyan cycle like the first unit.

I move it closer, restart it, no problems.

If I start the first Xenon (the one that began this thread) first and close to the Argon, it does the usual error sequence, but now the second unit will not connect anymore if I try and start it.

I suspect that there is something strange happening with the radio on the first Xenon that I enrolled that means 1. it does not connect properly or only rarely, and 2. it mucks up the mesh for the next Xenon.

Occasionally Xenon 2 gets into a rut (red and / or orange), which is fixed by shutting it and the Argon down, starting the Argon and then it.

[mstanley]

Hey Jmag,

That’s actually not unheard of. I have run into this incident before of a Xenon causing interference with the rest of the mesh network. While rare, I have seen it on one or two occasions.

The issues with the Argon and connectivity are consistent with the above Github issue. So that is something I suspect to be resolved in rc28.

For the Xenon having the radio connectivity issue, I’d highly recommend filing a support ticket

[jmag]

Hi Matthew,
I have tested it on a new network - new wifi, router, IP address, etc. and it does the same thing.

Once I turn on the problematic Xenon it fails to connect or connects after a long time and only temporarily. If the second Xenon that I setup is turned on first, it connects fast and works, until I connect the other one.

I have filed a support ticket, referencing this thread.

I am still having range anxiety with the ‘working’ xenon. When it is close to the Argon (<1m) it connects fast (<20sec), but by the time I get to 7m it begins to go through the flash cycle, connecting after about 1.5-2min.

Is this expected behaviour? Would it improve with an external antenna (once that feature is enabled in firmware - https://docs.particle.io/reference/device-os/firmware/xenon/#antenna-selection)?

Thanks for your persistence and help. John.

[mstanley]

Hey John, wanted to follow up with you on this. I know we put a request in to replace your Xenon device. Were you able to receive the new one and are you still experiencing any issues?

[jmag]

Hi Matthew, The replacement unit has not arrived yet. I am looking forward to testing it.

[kmmonk]

Hey @jmag - wanted to check-in on this, any update?

[jmag]

Hi, Sadly nothing yet.