Using Electron / Photon in high volume commercial product

[simmikolon]

Hi,

To keep it short: We (my company) want to integrate the Particle Electron / Photon (P1) into a commercial product. This product has an annual volume of 3000 units. We are however a bit concerned about reliability of those modules and the OTA-functionality of those particle modules. So I am having the following questions:

• Anyone here with high volume experiences / anyone who developed a product based on particle
• How reliable is the OTA-functionality - I remember some failed OTA during development forcing a flash via USB (which is unacceptable for our product)

Thanks,

Simon

[kennethlimcp]

@will @Charlie_K

Emailing sales@particle.io works too.

[simmikolon]

Thanks @kennethlimcp for letting me know but what I really want is feedback from an actual customer. I am pretty sure that no one from sales@particle.io will say that there is something wrong with using particle. Their answer would be something like “Yes of course you can do this! That’s what particle is made for!” - Well yes, definitely - but - I want real facts - not marketing speech from a sales guy …

But what about actual customers? Is someone out there or does someone know someone that uses particle modules in large fleets? I really would like to hear some feedback !!!

Take those guys from resin.io for example - they are making such a good job with their community and are posting use-cases / case-studies and customer stories. Something that particle does not so I am really afraid that particle.io is not a good idea for a large scale fleet commercial product!

I really do not want to think this but that’s what comes in mind when reading posts of the community.

And even if I might contact sales@particle.io -> I really want some “live” / in-field feedback from customers.

Thanks,

Simon

[Cameron]

Me too!

I’d like to see the lessons learnt that others might be able to factor in.

My two greatest fears are losing the APN setting or having the software lockup (has happened a few times to me) where I can only push new firmware by carefully timing a hardware reset with a new firmware push. (Tends to happen when over using delays, which I now avoid). The latter sadly doesn’t work when you put your device in a sealed enclosure with warranty stickers and they are located in remote areas!

[simmikolon]

@Cameron

The latter sadly doesn't work when you put your device in a sealed enclosure with warranty stickers and they are located in remote areas!

Hmm. I understand. How many of those sealed devices have been deployed? And what percentage of that number of devices has significant failure so that it cannot be re-activated / fixed by OTA? Do you have such numbers? Would be great to hear!

[Cameron]

Thankfully we haven’t deployed them yet - they are sitting ready to go next week after the firmware is made more robust.

At this stage - we were getting 25% failure rates on flash attempts. A lot of the time the devices said they were online, but were not. I suspected it was a issue with the connection keep alive time - but a whole lot of testing seemed to identify it wasn’t.

We have now put in more watchdog timers and I have noticed that at times I can still run a function… yet not be able to flash… So I have created a system.reset() function which lets us do the restart OTA which enables a flash. That seems to work 99% of the time - but we are still testing away to make sure.

Edit: I sometimes suspect a few of our functions (even with their delays or timers built in) when doing Particle.publish must clip the publish rate and maybe get blocked… but I don’t have a way to tell when a device of mine is blocked. Maybe that would be a great idea for the Particle dashboard to show when devices are blocked? At least would help with customer troubleshooting. We are lucky at the moment, I use a dashboard provided by Telstra (our local 4G provider) that lets us run a connection attempt all the way to the device at the network level to see if it is talking over TCP which helps identify if the device is at least powered on okay.

[simmikolon]

Oh wow.

So:

@Cameron

At this stage - we were getting 25% failure rates on flash attempts.

... means that a flash fails and that the device is dead and needs a wired re-flash? Or can you re-flash it using OTA?

I really appreciate your feedback @Cameron! Thanks!

BTW: We have tested 2 Electrons and 4 Photons in our industrial machines and those Electrons tend to have OTA flash issues ... more than those Photons ...

[sparkly]

Hi Simon,
Garadget (2K+ units deployed in a year):

My biggest lesson was that one shouldn’t do copper pour around P1 - PCB antenna doesn’t really appreciate it. The specs show exclusion zone directly under antenna, but keep it minimal even outside of it.

Also:

• There were no problems with mass OTA firmware deployment
• look into timers if you need a delay
• I wouldn’t keep APN on the device, you can manage that on the server end subscribing individual device IDs

[Cameron]

TBH, I also have a Arduino Nano in front of me that I was contemplating to use as a ‘fall back’ device that operates as an ‘offline’ version, similar capabilities - just minus a few online/SMS functions.

So I’d wire both devices in - but when the Particle device fails to kick in, in response to an input - the Micro would take over and shutdown the Particle via Relay to stop duplicate messages potentially coming out, take over - do it’s thing… then try to re-power the Electron.

However - at the moment that would require extensive logic circuits and PCB design - which I’m less confident about being highly reliable. That might be a different version of the product with a fail over design, but it would also double the cost of the product!

[Cameron]

@simmikolon

BTW: We have tested 2 Electrons and 4 Photons in our industrial machines and those Electrons tend to have OTA flash issues ... more than those Photons ...

Haven't had any issues with Photons, they seem pretty decent.

See also my post here where I've had to reflash via USB to bring them back alive again.

Electron: Error writing firmware - unknown message undefined

So after a good 4 hours today trying to get my Electron unbricked - I thought best to share my painful learning. I was getting the below message over and over, no matter what I tried, firmware updates, code version changes, DFU updates. [image] Error writing firmware - unknown message undefined In the end… the magic fix was via the particle-cli Step 1: Go to DFU mode (hold mode & reset, then release reset, and hold mode until the LED flashes yellow) Step 2: Run the command in terminal…

The 25% is OTA not working, but most of the time a power reset fixes it. Only a small percentage of the time do I need to do a re-flash.

The times when I need to do a re-flash my code via USB tends to have these three aspects:

1. Custom APN code use
2. Custom APN code was accidentally listed twice in code
3. Use of AT commands to communicate with the ublox module for SMS commands.

[Cameron]

@sparkly

I wouldn't keep APN on the device, you can manage that on the server end subscribing individual device IDs

Great insights, are you able to explain a little more by what you mean here?

Is that in reference to declaring the 'telstra.external' equivalent?
How do you do that in a way that the device can communicate before getting the APN details?

STARTUP(cellular_credentials_set("telstra.extranet", "", "", NULL));

[sparkly]

Sure, I assumed that APN in your post refers to Apple Push Notifications. If that’s the case, I’d suggest keeping the subscription records (Device ID to multiple push notification endpoints) on the server especially considering that the delivery of the notifications involves the use of server anyway.

[Cameron]

@sparkly

APN in your post refers to Apple Push Notifications.

Ahh - sorry. I was thinking APN as in the 3G internet gateway type service.

Access Point Name

Dang those similar acronyms!

[simmikolon]

@Cameron / @sparkly

Thanks for your feedback / insight - Would love to hear more.

Bit frustrating however that no official of @particle replies to my post. Hey folks at Particle take a look at how those guys at resin.io manage such questions - at least one official is always answering in a very short time!

[Cameron]

They are usually pretty good! It is a few days before easter - maybe quite a few are on holidays

Would be nice to see even more from them on collected insights from production grade/high volume users - we certainly need more of that ourselves!

[ScruffR]

@simmikolon

Bit frustrating however that no official of @particle replies to my post.

Hmm, that somehow contradicts this statement of yours earlier

@simmikolon

I really want is feedback from an actual customer. I am pretty sure that no one from sales@particle.io will say that there is something wrong with using particle. Their answer would be something like "Yes of course you can do this! That's what particle is made for!" - Well yes, definitely - but - I want real facts - not marketing speech from a sales guy ...

But as said: Holidays may get in the way - however these guys are so awesome that they may even respond in their free time - let's see what the next hours bring

[simmikolon]

@ScruffR

Hmm, that somehow contradicts this statement of yours earlier

Yepp - ok - That's true. To be honest - I really love the Electron - When comparing the BOM to parts directly ordered via Mouser / Digikey - even with a higher volume you can't get even close to the Electrons Pricing and that's why it it so damn interesting for us.

But we are talking about integrating this unit into 3k+ Devices and using its STM32 MCU as the main controller of those units. I have to make sure that those things will last in the next three + four years. And any details on current software quality - is greatly appreciated.

Well and yes -> even officials from particle could bring in more details from actual customers, for example.

But yes we'll see - and sorry if being impolite ...

Simon

[ScruffR]

No harm done, and I understand your point, hence the engagement

[developer_bt]

Each platform has its advantages and disadvantages.
However functionalities that have Particle platform is one of the best on the market.
The best is that the firmware is open source which helps to quickly and effectively detect and remove bugs together with the community.
It is based on Arduino / C / C ++, and also great is that there are many libraries that you can freely use.
There are other professional IoT platforms but none offers the opportunity for the rapid development of the product as Particle. Here you have the hardware, firmware and Cloud from one manufacturer.
However there is one problem, and that is the lack of support for TLS in case we want to connect with other Cloud services. When you enable this feature, Particle will definitely be a best platform for the development of IoT products.

When it comes to stability for Electron you can see code for fault tolerance example from @rickkas7.

Electron sample code for fault tolerance and problem debugging

Most of the time the Electron just works, happily connecting and disconnecting from the cellular network and all is good. Except once in a while, some devices have a hard time getting reconnected. Disconnecting the battery and power usually solves the problem, but that can be a pain for remote devices. This experiment is some code that might help recover from this situation automatically, and also log some data to better figure out what’s actually happening, so if it’s a bug, it can just be fix…

[Moors7]

@developer_bt

However there is one problem, and that is the lack of support for TLS in case we want to connect with other Cloud services.

There are webhooks that allow you to circumvent having to do heavy lifting on the device (as well as save on data). That said, there's work on TLS being done, so fingers crossed...