Photon web IDE "flash" stopped working [Resolved]

I and my partner have not done development for perhaps a month. Today when I got back into it, I found, and so did my partner, found that the flash function no longer works. Neither does the ‘safe mode’ button pressing sequence put the device into safe mode. I suspect this is a cloud issue.

Has anyone else seen this behaviour? Is it obvious what has gone south and how to fix it?

We have two separate development setups in two houses. Each of us has several Photons and it’s not working everywhere.

I note there is discussion in the documentation about keys and Cloud keys. But nowhere does it seem to say what these keys are. I raise this because perhaps some recent release of software at the Cloud end has caused these keys to get out of sync. This is a pure guess.

Our application runs in MANUAL mode and we have a keyboard command we can squirt in which tells the Photon to do a “Particle.connect();” Yes, of course, this is after wifi is established and, in fact, the Photon nicely communicates over a wifi socket to my Mac laptop. The base functionality is working fine; it’s just that we cannot flash new software in as of today.
–jim hahn

How exactly are you trying to get into safe mode? That has nothing to do with the cloud whatsoever and should thus work. There’s no reason why that would ‘suddenly’ stop working, let alone on different devices in multiple locations.

As far as not being able to flash goes, are you sure the device is connected properly before trying to do so (breathing cyan)?

You can also try putting it in DFU mode and flashing it over USB, which should always work :slight_smile:

2 Likes

Heureka, I found the exact location(s) of nowhere! :wink:
https://docs.particle.io/reference/cli/#particle-keys-doctor
https://docs.particle.io/reference/cli/#particle-keys-send
https://docs.particle.io/reference/cli/#particle-keys-server

BTW, some more info about your symptoms (e.g. what color codes does the device flash out) would be helpful.

2 Likes

Dear ScruffR and Moors7:
I had a bet with myself that you two people would reply. I win the bet. Thank you for being there for the community! I repeat my offer of a dinner and/or a beer if you get to Boston or Phoenix!

  1. Re ScruffR’s comment that he found ‘nowhere’! Let’s take a look, for example, at the very first lines of the ‘doctor’ reference:

Helps you update your keys, or recover your device when the keys on the server are out of sync with the keys on your device. The particle keys tools requires both dfu-util, and openssl to be installed.

My point is that it does not say what a key is used for. There is no concept of the basis for needing a ‘doctor’. This documentation note is useful only if you realize that keys can get out of sync. And under what conditions might you need to go to the doctor? What are the symptoms of a “bad key”?

So far I have not needed dfu-util although I knew it existed. And what is ‘openssl’? Remember: The particle products, like Photon, are intended for both hobbyist and serious sw development. As a hobbyist I am willing to learn this esoteric stuff but it would be nice to see some paragraphs on what the heck is going on at the big picture level.

  1. Is the device connected properly?

Yes, it’s connected exactly the same way it’s always been connected. It’s socketed in a small PC board surrounded by a bunch of glue logic for my application. I have six of these boards each with a Photon. They all cannot flash. I could have dropped a few brain cells (it happens) and not used “build” properly but my partner, over in his house with his own copies of the PC board, sees the same failure mode.

  1. Safe mode: Safe mode connects the Photon to the cloud, but does not run any application firmware.

When I said safe mode did not work, I meant that it did not connect to the cloud. In my mind, that’s it’s purpose: When a user app has a bug or something, you may need safe mode to get the device flashed with new user code so you can debug it.

  1. My user app. A few words.

The user app that’s currently in memory runs in MANUAL mode. That’s because our project is intended for use far away from civilization where there’s no Internet. Instead, we provide a local wifi net to allow many of our Photon based sensor boards to communicate with a host laptop. This works fine. We have to use MANUAL because, otherwise, the code will immediately hang looking for the real Internet, which ain’t around.

We have two ways to enter debug-like commands to the underlying user app. i) A serial port line. ii) From the laptop (which is running a server program), we can enter commands and they get squirted down to the Photon over a socket and the Photon’s reply appears on the laptop.

So how does one flash new code? If it’s MANUAL, there is no connectivity to the Internet. We have a command, named ‘cloud’, which commands our app to do a Particle.connect. After a one second delay, we check (using Particle.connected) if we have cloud connectivity and print out a message either way. The main loop of the app now knows it’s supposed to be looking for cloud connectivity and prints a message from that loop if it does NOT get connectivity. So that’s how I know that my app is not working. I get a string of “we are not cloud connected” messages. And so does my partner get the same messages.

Once we are ‘cloud connected’, a flash command from the build IDE can break through and stuff new user code into the Photon.

This has been working great for maybe 6 months.

If I know I am going to be doing a lot of edit/compile/flash/debug/curse iterations, I stick the Photon in AUTOMATIC mode to avoid all this rigamarole. Once I am happy, I do a final flash to put it back into MANUAL.

  1. LED colors

OK, so I plug my PC board in and it gets power. (Over USB cable.) My app starts. It blinks some lights on the PC board and uses the RGB LED to communicate its internal status. Looks good.

I issue one of my debugging commands to my app to tell me it’s detailed internal status. It tells me what wifi SSID it’s connected to and it’s the same as my laptop. It tells me it’s in MANUAL mode, wifi is “ready”, and we are NOT cloud connected. This looks fine

Now I want to flash and so I issue the ‘cloud’ command. As noted above, I get a message stream saying it cannot connect to the Cloud.

Time for “safe mode”!

Press Reset and Setup at same time. (BTW, documentation at one point refers to the Mode button which I assume is old nomenclature for the Setup button. At least, I hope so.)

RGB goes dark with both buttons pressed.

Release Reset.

Once you see magenta on the RGB, then release Setup.

I see maybe a few blinks of magenta (Yeah!) and then it goes to rapid blink cyan - “looking for cloud”. That’s maybe 15 seconds. Then it will flip to rapid blink green for a brief time - “connecting to wifi and looking for Internet”. Sometimes I see three longish red blinks. (I think they are red and not orange.)

Three red blinks from the documentation indicates: 3 red blinks: Connected to the internet, but could not reach the Particle Cloud. That is entirely consistent with my theory that cloud connectivity IS the problem. To say that differently, my app’s behaviour plus safe mode behaviour both point to yonder Cloud.

1 Like

The second and third link do state these

and

And this is a commonly used encryption scheme which can be found in loads of applications with plenty of resources to read up in more detail (if required)

https://docs.particle.io/support/troubleshooting/common-issues/photon/#blinking-cyan

Hence my question about your color codes.

OpenSSL and asymetric encryption is all but esoteric stuff - it is part of the framework our information-society/generation internet is built upon.

I guess this question was rather refering to the cloud connection and not to the electrical contacts - hence the term in parentheses

Yes, if your code was the problem, but since that does not work, Safe Mode has helped you to find out it's not your code :wink:
That's a valuable info too.

This is where particle flash --serial <yourNewFirmware.bin> or particle flash --usb <yourNewFirmware.bin> (requires dfu-util) come into the picture.

1 Like

Again, thanks for the info. I am still trying to come up to speed on all this.I am not going to fight you on what level of expertise a hobby user needs. (I’d only lose.)

Any idea how these keys could get out of sync simultaneously at two different development sites?

I shall dig into keys!
–jim

We don't intend to to fight either :smiley:
It's just that a lot of the things are there - even when not found.

And about the way how the keys can get out of sync, with devices not getting online for extended periods of times, the server keys might get changed several times in between but your device never gets these updates, or your public device keys might get "lost" as the device appears dead, or if your device gets into a brownout-condition which can damage the keys (this is a bug still under investigation), or other reasons.

Exactly for that time, the color codes provided by the RGB LED are important to provide to get some info about the actual reason why it cannot connect to the cloud.

You might see the same three red blinks as in Safe Mode, so that does indicate - as you suspected - a cloud handshake problem which might be cured with particle keys ...

1 Like

The saga continues. I am still down - cannot connect to cloud. Safe mode fails to connect.

ScruffR and Moors7 offered much advice. It seemed to come down to two ideas on the source of the problem.

  1. The “key” got corrupted or out of sync.
  2. The firmware in my Photons is corrupted or out of date.

I went through the process of generating a new key and automatically sending it to Particle by the scripts. The final output from my terminal session tells me this worked:
attempting to add a new public key for device 33002c000447343339373536
submitting public key succeeded!

One hole in the documentation is that you are supposed to email the new key to Particle. Alas, they don’t tell you WHERE to email it so that step is not done. But the “succeeded!” message above makes me think it will have no effect anyway. I emailed Particle from the Support Contact page asking to whom I should email my new key, but they have not replied. (Anybody know where you are supposed to email it?)

With respect to the firmware, I am currently running 0.5.2. Other than “prerelease” software, that seems to BE the current version. I don’t think my firmware is corrupted because simultaneous corruption on six Photons at once isn’t statistically possible.

So if anyone has suggestions, brilliant or otherwise (!), let me know. Maybe there’s a way to escalate this to the developers directly.

I can’t find an open support ticket for you, but in any case, it’s not necessary to email the keys anymore. All you should need to do is:

Put the Photon in DFU mode (blinking yellow) by holding down RESET and SETUP, releasing RESET and continuing to hold down SETUP while it blinks magenta until it blinks yellow, then release SETUP. Then issue the CLI commands:

particle keys doctor 33002c000447343339373536
particle keys server

Then reset your Photon and hopefully you’ll be back to breathing cyan.

Thank you for your advice but that did not work. (Groan.)

How does one open a support ticket? Yes, I expose ignorance…

ScruffR was working with someone a couple months ago on a similar problem of “cannot cloud connect”. At that time, they discovered a FAIL in the output of the ‘particle serial inspect’ command. Once that was fixed, the user was up and running. I ran that same command and there are no FAILs.

The symptoms seem to say that attempts to cloud connect from a Photon that is in MANUAL mode fails. No matter what. Safe mode also does not work.
–jim

Your Photon is definitely not connecting to the cloud; the last connection was 2016-08-16 at 00:09:06.682Z. I’m not sure why, though, if you have working Wi-Fi and you did the steps to reset the keys. That should have worked. I’ll need to see the whole log from the two commands; you can create a new support ticket at https://support.particle.io.

Sorry. I don’t see anything at “support.particle.io” related to opening a new support ticket. Or do you mean the big “Do you have a different question?” area? I have tried that before an never gotten anything back as a result.

Yes, the ask a different question area at the bottom of the support page. I’m not sure why it didn’t work for you, but you can also just send an email to hello@particle.io.

It’s there.

I’m sorry, I still can’t find your support ticket. There was a problem with the support portal earlier, but I think it has been fixed. We really are not actively trying to discard all of your support requests!

Understood. As you asked for the results of the two commands you wanted me to enter, and that’s what’s in the support ticket, let me copy them here also.
–jim

hahn:jimhahn jimhahn$ particle keys doctor 33002c000447343339373536
running dfu-util -l
Found DFU device 2b04:d006
running dfu-util -l
Found DFU device 2b04:d006
running openssl genrsa -out 33002c000447343339373536_rsa_new.pem 1024
running openssl rsa -in 33002c000447343339373536_rsa_new.pem -pubout -out 33002c000447343339373536_rsa_new.pub.pem
running openssl rsa -in 33002c000447343339373536_rsa_new.pem -outform DER -out 33002c000447343339373536_rsa_new.der
New Key Created!
running dfu-util -l
Found DFU device 2b04:d006
This file already exists, please specify a different file, or use the --force flag.
Continuing...
checking file 33002c000447343339373536_rsa_new.der
spawning dfu-util -d 2b04:d006 -a 1 -i 0 -s 34:leave -D 33002c000447343339373536_rsa_new.der
dfu-util 0.9

Copyright 2005-2009 Weston Schmidt, Harald Welte and OpenMoko Inc.
Copyright 2010-2016 Tormod Volden and Stefan Schmidt
This program is Free Software and has ABSOLUTELY NO WARRANTY
Please report bugs to http://sourceforge.net/p/dfu-util/tickets/

dfu-util: Invalid DFU suffix signature
dfu-util: A valid DFU suffix will be required in a future dfu-util release!!!
Opening DFU capable USB device...
ID 2b04:d006
Run-time device DFU version 011a
Claiming USB DFU Interface...
Setting Alternate Setting #1 ...
Determining device status: state = dfuIDLE, status = 0
dfuIDLE, continuing
DFU mode device DFU version 011a
Device returned transfer size 4096
DfuSe interface name: "DCT Flash   "
Downloading to address = 0x00000022, size = 610
Download	[=========================] 100%          610 bytes
Download done.
File downloaded successfully
Saved!
running openssl rsa -inform PEM -in 33002c000447343339373536_rsa_new.pub.pem -pubout -outform PEM -out /var/folders/rt/6wd_ghpx65vb9w9bd63l8vgh0000gn/T/116829-64195-ssjs9b.pub.pem
running openssl rsa -pubin -inform PEM -in 33002c000447343339373536_rsa_new.pub.pem -pubout -outform PEM -out /var/folders/rt/6wd_ghpx65vb9w9bd63l8vgh0000gn/T/116829-64195-ssjs9b.pub.pem
attempting to add a new public key for device 33002c000447343339373536
submitting public key succeeded!
Okay!  New keys in place, your device should restart.
hahn:jimhahn jimhahn$ particle identify

! serial: No devices available via serial
hahn:jimhahn jimhahn$ particle identify

Your device id is undefined
Unable to determine system firmware version
hahn:jimhahn jimhahn$ !p
particle identify

Your device id is 33002c000447343339373536
Your system firmware version is 0.5.2
hahn:jimhahn jimhahn$ particle keys server    OOPS, FORGOT TO PUT INTO DFU MODE.
running dfu-util -l

!!! I was unable to detect any devices in DFU mode...

> Your device will blink yellow when in DFU mode.
> If your device is not blinking yellow, please:

1) Press and hold both the RESET/RST and MODE/SETUP buttons simultaneously.

2) Release only the RESET/RST button while continuing to hold the MODE/SETUP button.

3) Release the MODE/SETUP button once the device begins to blink yellow.

Make sure your device is in DFU mode (blinking yellow), and is connected to your computer
Error - No DFU device found
hahn:jimhahn jimhahn$ !p          NOW IT'S IN DFU MODE
particle keys server
running dfu-util -l
Found DFU device 2b04:d006
checking file /usr/local/lib/node_modules/particle-cli/keys/rsa.pub.der
spawning dfu-util -d 2b04:d006 -a 1 -i 0 -s 2082 -D /usr/local/lib/node_modules/particle-cli/keys/rsa.pub.der
dfu-util 0.9

Copyright 2005-2009 Weston Schmidt, Harald Welte and OpenMoko Inc.
Copyright 2010-2016 Tormod Volden and Stefan Schmidt
This program is Free Software and has ABSOLUTELY NO WARRANTY
Please report bugs to http://sourceforge.net/p/dfu-util/tickets/

dfu-util: Invalid DFU suffix signature
dfu-util: A valid DFU suffix will be required in a future dfu-util release!!!
Opening DFU capable USB device...
ID 2b04:d006
Run-time device DFU version 011a
Claiming USB DFU Interface...
Setting Alternate Setting #1 ...
Determining device status: state = dfuIDLE, status = 0
dfuIDLE, continuing
DFU mode device DFU version 011a
Device returned transfer size 4096
DfuSe interface name: "DCT Flash   "
Downloading to address = 0x00000822, size = 512
Download	[=========================] 100%          512 bytes
Download done.
File downloaded successfully
Okay!  New keys in place, your device will not restart.
hahn:jimhahn jimhahn$

The steps you have done should have worked. I sent out a request to the rest of support to see if anyone else has ideas for things to try.

Thanks. Stay tuned for a higher level, new forum topic on “support” in general.

A developer checked the cloud logs for me and there’s no record of your device attempting to connect. Could you flash and run this program? You’ll need to flash it by USB, of course, and then capture the serial output from the program using particle serial monitor or a serial terminal program.

#include "Particle.h"

SYSTEM_THREAD(ENABLED);
SYSTEM_MODE(MANUAL);

//SerialDebugOutput debugOutput(9600, ALL_LEVEL);

void runWifiTests();
void runCloudTests();

const unsigned long INITIAL_WAIT_TIME = 5000;
const unsigned long RETRY_WAIT_TIME = 30000;
const unsigned long WIFI_CONNECT_WAIT_TIME = 120000;
const unsigned long CLOUD_CONNECT_WAIT_TIME = 120000;
const unsigned long CLOUD_PRE_WAIT_TIME = 5000;
const unsigned long CLOUD_POST_WAIT_TIME = 5000;


enum State { INITIAL_WAIT, WIFI_START, WIFI_WAIT_READY, WIFI_READY, WIFI_DISCONNECT,
	CLOUD_WAIT_CONNECTED, CLOUD_PRE_WAIT, CLOUD_CONNECTED, CLOUD_POST_WAIT,
	RETRY_WAIT };
State state = INITIAL_WAIT;
unsigned long stateTime = 0;

void setup() {
	Serial.begin(9600);
}

void loop() {
	switch(state) {
	case INITIAL_WAIT:
		if (millis() - stateTime >= INITIAL_WAIT_TIME) {
			// Wait 5 seconds before starting tests
			state = WIFI_START;
		}
		break;

	case WIFI_START:
		Serial.println("connecting wifi");
		WiFi.on();
		WiFi.connect();
		state = WIFI_WAIT_READY;
		stateTime = millis();
		break;

	case WIFI_WAIT_READY:
		if (WiFi.listening()) {
			Serial.println("entered listening mode");
			state = WIFI_DISCONNECT;
		}
		else
		if (WiFi.ready()) {
			Serial.println("connected to Wifi");
			state = WIFI_READY;
		}
		else {
			if (millis() - stateTime >= WIFI_CONNECT_WAIT_TIME) {
				Serial.println("failed to connect to Wifi");
				state = WIFI_DISCONNECT;
			}
		}
		break;

	case WIFI_READY:
		runWifiTests();
		Particle.connect();
		state = CLOUD_WAIT_CONNECTED;
		stateTime = millis();
		break;

	case CLOUD_WAIT_CONNECTED:
		if (Particle.connected()) {
			Serial.println("connected to the cloud!");
			state = CLOUD_PRE_WAIT;
			stateTime = millis();
		}
		else {
			if (millis() - stateTime >= CLOUD_CONNECT_WAIT_TIME) {
				Serial.println("failed to connect to the cloud");
				Particle.disconnect();
				state = WIFI_DISCONNECT;
			}
		}
		break;

	case CLOUD_PRE_WAIT:
		if (millis() - stateTime >= CLOUD_PRE_WAIT_TIME) {
			state = CLOUD_CONNECTED;
		}
		break;

	case CLOUD_CONNECTED:
		runCloudTests();

		state = CLOUD_POST_WAIT;
		stateTime = millis();
		break;

	case CLOUD_POST_WAIT:
		if (millis() - stateTime >= CLOUD_POST_WAIT_TIME) {
			Serial.println("disconnecting from cloud");
			Particle.disconnect();
			state = WIFI_DISCONNECT;
		}
		break;

	case WIFI_DISCONNECT:
		WiFi.disconnect();
		WiFi.off();
		state = RETRY_WAIT;
		stateTime = millis();
		break;

	case RETRY_WAIT:
		if (millis() - stateTime >= RETRY_WAIT_TIME) {
			state = WIFI_START;
		}
		break;
	}
}

void runWifiTests() {
	IPAddress addr = IPAddress(8,8,8,8);
	int res = WiFi.ping(addr, 1);
	Serial.printlnf("ping 8.8.8.8 %s", (res == 1) ? "success" : "failure");

	addr = WiFi.resolve("api.particle.io");
	if (addr) {
		Serial.printlnf("api.particle.io = %s", addr.toString().c_str());
	}
	else {
		Serial.println("failed to resolve api address");
	}

	addr = WiFi.resolve("device.spark.io");
	if (addr) {
		Serial.printlnf("device.spark.io = %s", addr.toString().c_str());

		TCPClient client;
		if (client.connect(addr, 5683)) {
			Serial.println("connected to device server CoAP!");
			client.stop();
		}
		else {
			Serial.println("could not connect to device server by CoAP");
		}
	}
	else {
		Serial.println("failed to resolve device address");
	}


}

void runCloudTests() {
	Particle.publish("apiServerCheck", "passed", PRIVATE);
}

Thanks,
Rick

OK. But first I have to go climb Mount Everest. I have never flashed from USB. I guess that means an offline compile/link and then download? I will start hunting how to do that but if you have a pointer to a place to start, that would be great.

We had hoped never to have to use USB flash and all. But I understand the need….