I am using a P1 in a PCB that we have designed in-house.
I have recently started updating our fleet from Particle OS 0.6.3 to 1.4.4.
Does anyone have experience with a failed OS upgrade that causes listening mode to fail in safe mode?
Summary
There are 2 instances where the user has lost power before the upgrade process has been fully completed. This has resulted in an almost entire loss of system control (listening mode included).
I have placed the boards in safe mode so that I could attempt another flash but have lost the ability to connect to WiFi.
Things we’ve tried
Safe mode: The user is able to place the board in safe mode (breathing magenta)
Listening mode (in safe mode): The user is unable to initiate listening mode, setup button is non-responsive (blinking blue)
Things we’ve discovered
When entering safe mode, the LED goes straight to magenta and does not cycle through the typical green -> cyan cycle before fully landing in safe mode.
From safe mode we cannot seem to enter listening mode (holding the setup button for ~3 seconds). It stays magenta the entire time and does not respond when setup is pressed.
Reset does pull the unit out of safe mode and back into the critically impacted OS.
Things we are going to try
DFU mode: I will receive these boards in the coming weeks and will perform a serial recovery process.
Questions
Is there anything I can try to allow me to recover from this broken safe-mode state in the field?
My apologies if this is a trivial question, but have you tried forcing Listen Mode by clearing the WiFi Credentials??
I am a bit uncertain about the state of your buttons, you mention the user can get it into 'Safe Mode' all be it non-responsive. Can the user use the buttons to clear the WiFi Creds then? This should force the device into Listen mode.
Having said this, getting the device into Listen Mode might not resolve any bootloader or Operating system file issues? Of course if it does connect to WiFi successfully afterwards you can try OTA and re-flashing to see if it resolves any issues.
I will attempt this with the user and report back.
Regarding the state of my buttons:
Initiating Safe mode from normal operation (setup + reset): works
Reset from Safe mode (reset): works
Initiating Listening mode (setup): does not work in either safe mode nor under normal operation
Reset from normal operation (reset): cannot determine (LED is the only indicator and it is off)
Reset wifi credentials from normal operation (setup): cannot determine (LED is the only indicator if this has worked)
Reset WiFi credentials from safe mode (setup): *will attempt and report back
My hunch is that the buttons themselves can correctly interrupt, however the majority of functions/variables they are calling are missing in both operating states.
Well, my guess would be that if Clear WiFi Creds worked and you restart (or even without restart) the device it would have entered Listen Mode. So if you press+hold Setup for around 15 seconds, and you do not pick up the device from a mobile phone or some other device, I am fairly sure the Clear Credentials failed.
Seems likely to be the case then yes. Interrupts should work regardless of the state. I have interrupts calling ListenMode and they work regardless, that being said, IF your code is 'accessible'.
It seems DFU mode might to be your Hail Mary here. Do you have USB access?
I do but not necessarily with interrupting OS update, but I managed to completely confuse a Photon by removing power during some critical function. See this post.
I threw everything I had at it, but I have not been able to restore it even using the debug shield I would not expect such dire consequences from a failed OS update though. There are some really clever guys in here, maybe someone else can shed more light. I will dig a little deeper, if I find something I will let you know.
Do you have the RGB LED wired up as per the Datasheet? If so I my understanding of the data sheets is that I will indicate Listen Mode if the buttons are successful to enter Listen Mode or forcing Listen Mode by means or clearing credentials?
If it is not indicating Listen Mode, I think it is safe to assume Listen Mode is not working. If it was me, I would start by trying to re-flash all files from scratch.
There are links to P1 bootloader downloads should you require them.
Thanks for this tip, I'm going to try this with the user tonight.
I do not have access in the field. Once I receive this board back from the user I can attempt a DFU recovery.
Thanks, I'll take a look through here to see if there are any similar occurrences!
We've wired the majority of the LED colors as per the datasheet with only a few minor changes. The only LED pattern that I'm able to get is magenta from initiating safe mode otherwise it is off.
We were unsuccessful in our attempt to wipe WiFi credentials while under normal operation and safe mode. We monitored networks available while performing these tests and and the SSID did not appear at any point.
I’ll hopefully be able to learn more once I receive the boards back.
Thanks for your help, I’ll keep this post up-to-date with anything I discover.
I cannot think of a way to give USB access to current users without granting access to our Particle credentials, which are a requirement for Particle CLI usage. Functions like particle flash --usb xxx.bin and particle keys doctor won't work without being logged in.
I wouldn't mind mind a scenario where the user connects their computer to the device via USB and I remote desktop their machine as a debugging proxy. However I still cannot think of a way to get their local machine to perform a particle flash --usb without logging in locally.
Are you aware of any secure methods to debug usb remotely?
My apologies for not being clear, this is what I meant yes. I learned not to assume that people do have access as I have dealt with someone not too log ago that deviced it was worth saving the ±$1.00 and not install USB ports on the PCB
Hhmmm.... interesting question, let me see what I can find out. There is a fantastic product called LogMeIn that we used extensively for all sorts of remote support. This should allow you to use the client PC as a gateway and transfer files back and forth and flash the devices accordingly. It has been a while since I have worked on this so I am not100% sure, but it might be an option if this is something you encounter frequently.
Have you played around with 'customers', 'authentication' or maybe even team options Particle Console? It might also be feasible adding this client's devices under a product and assign a team member that will then have access to their devices only. It should also then allow them to flash to those devices??
You can also force listening mode and DFU through the CLI using the magic baud rates, 28800 and 14400 respectively, and I don’t believe you have to be logged in for this.
@ScruffR Thanks for the correction! I’m not sure why I assumed this. Just tested locally and you are correct, you do not need to be logged in.
With this I can remote desktop their local machine, install particle CLI + dependencies, then debug. I’ll report back if I run into any issues along the way.
I may consider creating a Docker image if this becomes common (Docker Doctor lol). It could include libraries, bin files and reporting functions to streamline the process.
Totally forgot about this! We are enabling DFU by setting baud rates for an automated manufacturing process. I should have known to check for listening mode baud rates!
I could have used this method to test the severity of our listening mode failure before having the board shipped back.
I will definitely keep this in mind for next time, thanks for the tip!
Which is why I suggested LMI Rescue (or of course something similar of course) When we used it, it would i.e give us access directly to the device connected to client PC from a local console without have to Remote Desktop to the clients pc. Was couple of years ago, so might have changed and was quite pricey to be honest, but worth it if you do a lot of remote support.
To confirm what @ScruffR said, I tested on a Photon quick, seems to flash just fine. I normally use Workbench CLI so never really paid attention whether to I am logged in or not Good to know.
I would not expect such dire consequences from a failed OS update though. There are some really clever guys in here, maybe someone else can shed more light. I will dig a little deeper, if I find something I will let you know.
When we used it, it would i.e give us access directly to the device connected to client PC from a local console without have to Remote Desktop to the clients pc. Was couple of years ago, so might have changed and was quite pricey to be honest, but worth it if you do a lot of remote support.
Good to know.