2019 priorities for 3rd Gen hardware stability

I didn’t answer the general feedback portion of the question. I’ve seen much-improved stability on RC27… but not perfect. Good job getting to this point! My heartbeat code generally is at something like 99% reliability. I have the Argon gateway and all 4 xenon nodes sitting on my desk not even 18 inches apart all plugged into my PC and I’m still getting “lost” messages from one endpoint on a regular basis. The heartbeat from the Argon goes out every 10 seconds to which all 4 Xenon endpoints respond with their device ID. The endpoint that doesn’t respond isn’t always the same. In the screenshot of my Losant dashboard, I added blue tick marks on the X axis at all the places I can see where a heartbeat response is missed. Over the 6 hours of this graph, there are 8 missing responses: 6 hrs * 60 minutes/hr * 6 pubs/minute = 2160 total pubs. 2152 responses / 2160 total pubs = 99.6% reliability.

In comparison, I have another mesh network at home with an Argon gateway and a single Xenon endpoint. That network is rock-solid and never misses a heartbeat response:

One possible explanation for the missing messages is that 2 response messages arrive at the exact same time. I wrote the subscribe callback function to record data and exit as quickly as possible so I would think it’s something else. I just don’t know how to diagnose for you Particle experts any further.

However, on my 2-device home network, I have noticed several times that my gateway will go into blinking green and is no longer publishing to the cloud. The Xenon endpoint also goes into blinking green because the gateway’s state. Resetting the Argon gateway will get back to breathing cyan almost immediately. I have left the Argon in blinking green for 2 days and it never self recovered. It required a press of reset to recover and goes to breathing cyan almost immediately. It has done this at least twice on RC27. These nodes are simply inserted in the stock breadboard, powered via USB via a wall charger, 500mAh LiPo attached to LiPo connector, no external sensors or wiring attached and left with the heartbeat code running. I cannot correlate the drops in connectivity to anything environmental, WiFi drops, etc. Seems random. I wouldn’t put it past Verizon FIOS to have a network outage when these drops happen but that’s just speculative.

2 Likes

@ninjatill, I have a 5-node Argon/Xenon mesh running at home. Three nodes are running the latest Marco-Polo with the Argon, another is firing a webhook to fetch weather data and then Mesh pubs the “massaged” data to the last node which displays it on a Waveshare eink display. One of the Marco-polo nodes is 35ft away from the Argon (with an interior wall in between) and both the Argon and that node have Mesh antennas connected. I haven’t got the data going to Losant as you do (didn’t have the time to setup Losant).

I often see one node missing a reporting cycle and I suspect that is due to “collisions” on the UDP socket at the receiving Argon. Since UDP doesn’t retry, the data simply gets lost.

One way around this is a short random delay on the node response. Another is for the Argon to publish an ACK message that the node subscribes to. I would recommend this approach to implement no-fail messaging on the mesh. However, it should only be used for no-fail type data. Again, good mesh planning is key here.

I suggest this discussion be taken to another topic to avoid hijacking of the original topic.

3 Likes
  1. Sleep mode(s) especially for the Boron. I’m impressed so far with Mesh devices low power usage compared to Gen2, but the Boron being the mobility device of choice with a power hungry cellular module, we really need at least one sleep mode that will put that module to sleep, and ideally wake it up by either a pin trigger or a cellular ping just like Electron 3G.

  2. Update particle CLI to manage mesh networks: create, delete, add node/gateway/repeater, check status via serial communication or BLE.

3 Likes

Tools to show the topology of the network. I want to be able to see where my single points of failure are in the mesh.

3 Likes

My personal priorities are for…

  1. Hardware watchdog
  2. 2MB flash memory access
  3. HA networks. I’m likely to have multiple argons in close proximity and would love for them to programmatically use the local mesh comms to get online in case one of their ethernet cables gets unplugged

A post was split to a new topic: Xenons having trouble joining Argon Mesh

  1. Mesh topology/debugger tool in the form of a tutorial, so it can be used when setting up the network. It could be an extension of Tinker, but also a project to be loaded by a first time user. It would help debug setup issues. Examples are the Marco Polo code (@ninjatill) and the Mesh Hello World (@rocksetta).
  2. Fully functional debugging in Workbench… traps, run to, show variables, send values to published functions…
  3. I don’t know if this would fall under the hardware watchdog, but it would be awesome if something like the SparkIntervalTimer functionality (or a subset) is built into the DeviceOS. For Photons and Electrons as well.
    It would simplify coding for applications with IR control, ultrasound range sensors, LoRa…
  4. BLE support. I wish the Gen3 hardware could be the gateway for my BluzDKs, even without revealing the BluzDK on the cloud. At least I can pass data back and forth.

+1 for the BLE API
I have a couple projects that would definitely benefit from this.

1 Like

What I’d love to see next:

  • sleep modes
  • BLE API

Thanks for asking!

Wanted to circle back to thank everyone for the input! We see some clear patterns emerging and will take this input into our sprintly product planning process. We have a couple of more near-term stability fixes out the door, and hope to get to some of the juicy new product features (Bluetooth, sleep modes, HA networks) shortly.

7 Likes

For me, I’d love to see items in the following order:

  • BLE API
  • NFC API
  • Allow devices of all sorts to join the same mesh. No reason why two Argons can’t join the same mesh (even if only one is the ‘master’).
  • Mesh Network being able to use alternative gateways. For example, if I have 3 Xenons, 1 Argon and 1 Boron – if the WiFi goes down, the devices connect to the cloud via the Boron, and switch back if the WiFi becomes stable again.
  • Easier way to have Argons switch WiFi networks. Easier for units to switch the Mesh they are apart of.
  • Support for the debugger in the VS-Code environment.

BLE API
(Assuming this means directly communicating with a xenon / boron / argon from a standalone BLE device)

  • I, and many others, want to be able to connect directly to these devices with our own custom apps on our smartphones to interface with them.

I can see so many applications where for less than $10, you can add configurability through a smartphone. The possibilities seem endless. Industrial sensor configuration? Screw you HART foundation. I have BLE on my sensors. It doesn’t end there… #limitless

5 Likes
  • XIP access for more program space would be a huge reason to switch from Photon to Mesh devices for production products
  • BLE would awesome

^ THIS ^

I kind of doubt we'll see this happen (for various reasons, some practical, some business/ROI-related), but I'd love to be wrong! Edit: Okay, reserving judgement until HA details come out. :slight_smile:

4 Likes

@dougal, this functionality is exactly what is being planned for the High Availability (higher cost tier) mesh networks.

So you’ll be able to specify “primary” and “secondary” gateways? I had the impression that the mesh would just always use the “best available” gateway, based on nearest path. If you can somehow configure (or if it’s just automatic) that WiFi should always be primary and only use Cellular when no WiFi gateways are available, that will be better than what I was expecting.

@dougal, how Particle implements HA is still not defined. The “best available” may be based on “cost of route” or pre-defined fallback order. In a true HA mesh, you may have primary and secondary gateways in different segments of a large mesh (eg. “at both ends”). There are many possibilities which Particle needs to explore.

3 Likes

I like @peekay123’s list + HA capability as already discussed.

If I pick up one item, it’s stability – the mesh stability of course, and the cloud connection stability also. I think the cloud integration is a major differentiator of Particle Mesh, but I am disappointed and frustrated when the device seems disconnected from the cloud so often (i.e. Particle.publish/Ping/Diagnostic/OTA etc. do not go through) although I am sure the device is sending the events over the local mesh network (by Mesh.publish) perfectly fine.

  • General reliability
  • Mesh networks
  • WiFi connectivity
  • cloud monitoring of devices
  • NFC API

Thanks for asking.

Thanks all for taking the time to share this feedback. It all aligns pretty well with our expectations. For those counting at home, here's the feedback tally (below).

We've already begun preliminary work on the first two (BLE APIs and Sleep modes) and will begin discussion of the features below internally.

BLE API ++++++++++++
Sleep modes +++++++++
Device recovery (HW watchdog) +++++
NFC API ++++
Topology mapper ++++
User access to external flash +++
HW debugging in workbench +++
High availability networks +++
Mesh management API +
One free HA network +
Stable threading
Sleepy end devices
"Efficient" mesh OTA (download once, distribute locally)
Boron Cellular API completeness
Cellular data metering in dashboard
Update CLI to support Mesh
Gateway can join mesh as an end device
Easier switching of Wi-Fi / mesh networks
XIP for external flash

9 Likes