What would cause a device to be online sending data every 5 minutes like clockwork (using Particle.Publish) but unable to call functions, unable to take vitals, unable to load new software, even unable to ping it? It seems like it's only a one way communication (Device can send data to the cloud, but the cloud is unable to send data/communicate to the device). Device is on 5.5.0 and I have several hundred devices running the same code so I don't think it has anything to do with that. I did cycle power to it and that seemed to work for about 5-10 minutes after it connected but then the same behavior.
What is even more strange is I am using a watchdog on the device and the only time the watchdog gets pet is a successful ACK from a particle.publish() event. It doesn't seem to be resetting at all so it's like the device is getting the ACK from publish events but that is it.
This souds very much like your code may be starving the cloud process from processing time.
Not sure why you see that behaviour only on one of your devices though.
Your Particle.publish() calls work since your device is actively sending data to the (permanently) listening cloud.
On the other hand, in order to receive communication from the cloud your device must attend to such requests there and then - near instantly.
If your code has long running, blocking sections your device may be missing the requests.
The easiest way to deal with that is by means of using SYSTEM_THREAD(ENABLED).
Alternatively you could actively instruct the device from your code to regularly check for any pending requests by calling Particle.process() which usually gets called implicitly between each iteration of loop() and once per accummulated second of delay() time.
It's also good practice to avoid any code blocks that keep the process trapped for extended periods, wherever possible.
@ScruffR - Yeah that all makes sense. I am using System_Thread(Enabled) and anytime I use a delay, I implement the "SoftDelay" where it keeps calling particle.process() so I am not aware of any circumstance where my main loop blocks for extended periods of time. I also have this same code on hundreds of devices but the issue seems to be isolated to a single device. This device does occasionally flip flop between AT&T coverage and Verizon service.
I'm not sure if it's trying to send the command to the device over the AT&T towers while it's connected to Verizon or what the issue could be?