DeviceOs 2.2.0-rc2 seems to toast softAP connection to cloud

Firmware Libraries
1

Original title:
SoftAP sets WI-FI, Photon breaths green BUT it is connected to cloud

PROBLEM

I am using the SoftAPLib library (the one present in the libraries at build.particle.io).
After settings wifi credentials via SoftAP, my photon either breaths green (as in Cloud not connected mode), or it goes back to blinking blue (as in listening mode).
BUT when it is breathing green, the photon is indeed connected to the cloud as I can signal it from the particle console and it shoots rainbows at me.

HOW TO REPRODUCE

I flash this firmware on my photon:

#include <SoftAPLib.h>
SerialLogHandler logHandler(LOG_LEVEL_ALL);
STARTUP(softap_set_application_page_handler(SoftAPLib::getPage, nullptr));
SYSTEM_THREAD(ENABLED);
unsigned long logTime = 0;
void setup()
{
}
void loop()
{
  if (millis() - logTime > 5000)
  {
    logTime = millis();
    Log.info("Hello SoftAP from loop()!");
  }
}

I start on a photon by deleting ALL wifi credentials (looong press on the MODE button until it flashes blue few times).
If I reset it, it blinks blue as in listening mode. All good.

Now I connect to it with my computer, scan and set the wifi credentials via the softAP pages.
The device reboots and it:

In this state, from the particle console, I can run vitals on it and I can signal it (it shoots rainbows on the LED).

If I reset it, it either:

Logs

Here’s a log from when it ends up blinking green:

$ particle serial monitor --follow

Serial monitor opened successfully:
nchangEnter 63-digit claim code: FG?0000030006 [app] INFO: Hello SoftAP from loop()!
0000040008 [app] INFO: Hello SoftAP from loop()!
0000045009 [app] INFO: Hello SoftAP from loop()!
0000050010 [app] INFO: Hello SoftAP from loop()!
<><><> <><><>
<><><> <><><> this is when I connect my computer to the photon softAP and load 192.168.0.1 on a browser
<><><> <><><>
handling page /index
sending redirect
handling page /index.html
handling page /style.css
handling page /rsa-utils/jsbn_1.js
handling page /rsa-utils/jsbn_2.js
handling page /rsa-utils/prng4.js
handling page /rsa-utils/rng.js
handling page /rsa-utils/rsa.js
handling page /script.js
handling page /favicon.ico
0000160032 [app] INFO: Hello SoftAP from loop()!
0000165033 [app] INFO: Hello SoftAP from loop()!
<><><> <><><>
<><><> <><><> here I clicked on set wifi creds on the softAP page on my browser
<><><> <><><>
0000168945 [system] INFO: Cloud: connecting
0000169017 [system] INFO: Cloud socket connected
0000169019 [comm.protocol.handshake] INFO: Establish secure connection
0000170034 [app] INFO: Hello SoftAP from loop()!
0000170131 [comm.protocol.handshake] INFO: Sending HELLO message
0000170132 [comm.protocol.handshake] INFO: Receiving HELLO response
0000170132 [comm.protocol] INFO: waiting 4 seconds for message type=0
0000170733 [comm.protocol] TRACE: rcv'd message type=0
0000170733 [comm.protocol.handshake] INFO: Handshake completed
0000170761 [comm.protocol] INFO: Posting 'S' describe message
0000170787 [comm.protocol] INFO: Posting 'A' describe message
0000170788 [comm.protocol] INFO: Sending subscriptions
0000170892 [system] INFO: Cloud connected
0000170996 [comm.protocol] TRACE: Reply recieved: type=2, code=0
0000170996 [comm.protocol] TRACE: message id 10437 complete with code 0.00
0000170996 [comm.protocol] TRACE: rcv'd message type=13
0000171099 [comm.protocol] TRACE: Reply recieved: type=2, code=69
0000171099 [comm.protocol] TRACE: message id 10443 complete with code 2.05
0000171099 [comm.protocol] INFO: Received TIME response: 1645649134
0000171101 [comm.protocol] TRACE: rcv'd message type=12
0000171202 [comm.protocol] TRACE: Reply recieved: type=2, code=0
0000171202 [comm.protocol] TRACE: message id 10444 complete with code 0.00
0000171202 [comm.protocol] TRACE: rcv'd message type=13
0000171303 [comm.protocol] TRACE: Reply recieved: type=2, code=0
0000171303 [comm.protocol] TRACE: message id 10445 complete with code 0.00
0000171303 [comm.protocol] TRACE: rcv'd message type=13
0000171405 [comm.protocol] TRACE: Reply recieved: type=2, code=0
0000171405 [comm.protocol] TRACE: message id 10446 complete with code 0.00
0000171405 [comm.protocol] TRACE: rcv'd message type=13
0000172617 [comm.protocol] INFO: Received DESCRIBE request; flags: 0x04
0000172618 [comm.protocol] INFO: Posting 'M' describe message
0000172621 [comm.protocol] TRACE: rcv'd message type=1
0000172722 [comm.protocol] TRACE: rcv'd message type=8
0000172823 [comm.protocol] TRACE: Reply recieved: type=2, code=0
0000172823 [comm.protocol] TRACE: message id 10447 complete with code 0.00
0000172823 [comm.protocol] TRACE: rcv'd message type=13
0000175035 [app] INFO: Hello SoftAP from loop()!
0000180036 [app] INFO: Hello SoftAP from loop()!
0000185037 [app] INFO: Hello SoftAP from loop()!
0000188077 [comm.protocol] TRACE: Reply recieved: type=2, code=0
0000188077 [comm.protocol] TRACE: message id 10448 complete with code 0.00
0000188077 [comm.protocol] TRACE: rcv'd message type=13
0000190038 [app] INFO: Hello SoftAP from loop()!
0000195039 [app] INFO: Hello SoftAP from loop()!
0000200040 [app] INFO: Hello SoftAP from loop()!
0000203230 [comm.protocol] TRACE: Reply recieved: type=2, code=0
0000203230 [comm.protocol] TRACE: message id 10449 complete with code 0.00
0000203232 [comm.protocol] TRACE: rcv'd message type=13
<><><> <><><>
<><><> <><><> here the photon is breathing green (as in "Cloud not connected mode"), from the particle console online I can signal it and it shoots rainbows at me
<><><> <><><>
<><><> <><><>
<><><> <><><> the photon is executing the loop() function as I can see these "Hello SofAP" messages come out
<><><> <><><>
0000561608 [comm.protocol] TRACE: Reply recieved: type=2, code=0
0000561608 [comm.protocol] TRACE: message id 10474 complete with code 0.00
0000561610 [comm.protocol] TRACE: rcv'd message type=13
0000565113 [app] INFO: Hello SoftAP from loop()!
0000570114 [app] INFO: Hello SoftAP from loop()!
0000575115 [app] INFO: Hello SoftAP from loop()!
0000576761 [comm.protocol] TRACE: Reply recieved: type=2, code=0
0000576761 [comm.protocol] TRACE: message id 10475 complete with code 0.00
0000576761 [comm.protocol] TRACE: rcv'd message type=13
0000580116 [app] INFO: Hello SoftAP from loop()!
0000585117 [app] INFO: Hello SoftAP from loop()!
0000590118 [app] INFO: Hello SoftAP from loop()!
0000592013 [comm.protocol] TRACE: Reply recieved: type=2, code=0
0000592013 [comm.protocol] TRACE: message id 10476 complete with code 0.00
0000592013 [comm.protocol] TRACE: rcv'd message type=13
0000595119 [app] INFO: Hello SoftAP from loop()!

Recovery steps

When it goes in the stubborn blinking blue state, one way to get out of it was to flash tinker, then flash this firmware again. I heard this happened to another particle user and that’s the way they used to get out of it.

Other posts where a similar issue is reported

Here → no conclusion.
and here

Impact

As far as I know, this happened to at least 3 different photons in 3 different locations with 3 different Particle accounts, 2 different countries, only one continent, across two different products, and a personal account, and on 3 different firmware.
This leads me to believe this is not a router problem.
I also believe it is not a firmware problem (example: something blocking loop() ).
But it could be a library problem or a DeviceOS problem. Or something else that I’m missing.

Other steps

I’d try to bastardize the Particle Photon Wi-Fi debugging tool from @rickkas7 and mix it with this softAP firmware to see if I can run it when the photon is breathing green.

UPDATE:
I created a softap firmware with wifi debug included and it is here.

Logs:

<><><> <><><>
<><><> <><><> photon starts in listening mode, blinking blue, wifi credentials reset by me
<><><> <><><>
available access points:
SSID=randomwifi1 security=unsecured channel=11 rssi=-66
SSID=myrouter security=wpa2 channel=11 rssi=-61
SSID=VIRGIN903 security=wpa2 channel=1 rssi=-71
SSID=myrouter-guest security=wpa2 channel=11 rssi=-59
SSID=Habs security=wpa2 channel=6 rssi=-66
SSID=DIRECT-A6-HP OfficeJet Pro 8710 security=wpa2 channel=11 rssi=-78
SSID=RandomDeco security=wpa2 channel=3 rssi=-69
available access points:
SSID=myrouter security=wpa2 channel=11 rssi=-51
SSID=randomwifi1 security=unsecured channel=11 rssi=-62
SSID=myrouter-guest security=wpa2 channel=11 rssi=-51
SSID=DIRECT-A6-HP OfficeJet Pro 8710 security=wpa2 channel=11 rssi=-71
SSID=Habs security=wpa2 channel=6 rssi=-64
SSID=RandomDeco security=wpa2 channel=3 rssi=-75
SSID=RandomDeco security=wpa2 channel=3 rssi=-73
SSID= security=wpa2 channel=3 rssi=-73

<><><> <><><>
<><><> <><><> I am connecting and set wifi creds with softAP
<><><> <><><>
available access points:
SSID=DIRECT-A6-HP OfficeJet Pro 8710 security=wpa2 channel=11 rssi=-75
SSID=myrouter security=wpa2 channel=11 rssi=-49
SSID=randomwifi1 security=unsecured channel=11 rssi=-69
SSID=myrouter-guest security=wpa2 channel=11 rssi=-50
SSID=Habs security=wpa2 channel=6 rssi=-66
SSID=RandomDeco security=wpa2 channel=3 rssi=-72
SSID=RandomDeco security=wpa2 channel=3 rssi=-69
SSID= security=wpa2 channel=3 rssi=-74
handling page /index
sending redirect
handling page /index.html
handling page /style.css
handling page /rsa-utils/jsbn_1.js
handling page /rsa-utils/jsbn_2.js
handling page /rsa-utils/prng4.js
handling page /rsa-utils/rng.js
handling page /rsa-utils/rsa.js
handling page /script.js
available access points:
SSID=DIRECT-A6-HP OfficeJet Pro 8710 security=wpa2 channel=11 rssi=-73
SSID=myrouter security=wpa2 channel=11 rssi=-67
SSID=myrouter-guest security=wpa2 channel=11 rssi=-50
SSID=randomwifi1 security=unsecured channel=11 rssi=-64
SSID=Habs security=wpa2 channel=6 rssi=-65
<><><> <><><>
<><><> <><><> here the photon is breathing green (as in "Cloud not connected mode"), from the particle console online I can signal it and it shoots rainbows at me
<><><> <><><>
available access points:
SSID=myrouter-guest security=wpa2 channel=11 rssi=-71
SSID=randomwifi1 security=unsecured channel=11 rssi=-69
SSID=myrouter security=wpa2 channel=11 rssi=-68
SSID=Habs security=wpa2 channel=6 rssi=-76
0000190776 [app] INFO: Hello SoftAP from loop()!
0000195777 [app] INFO: Hello SoftAP from loop()!
configured credentials:
ssid=myrouter security=wpa2 cipher=1
available access points:
SSID=myrouter security=wpa2 channel=11 rssi=-66
SSID=randomwifi1 security=unsecured channel=11 rssi=-69
SSID=myrouter-guest security=wpa2 channel=11 rssi=-67
SSID=Habs security=wpa2 channel=6 rssi=-77
0000200778 [app] INFO: Hello SoftAP from loop()!
0000205779 [app] INFO: Hello SoftAP from loop()!
<><><> <><><>
<><><> <><><>  if I reset it: breaths blue ()
<><><> <><><>
Serial connection closed.  Attempting to reconnect...
Serial monitor opened successfully:
configured credentials:
ssid=myrouter security=wpa2 cipher=1
available access points:
SSID=myrouter-guest security=wpa2 channel=11 rssi=-67
SSID=randomwifi1 security=unsecured channel=11 rssi=-71
SSID=myrouter security=wpa2 channel=11 rssi=-69
0000030763 [app] INFO: Hello SoftAP from loop()!
0000035764 [app] INFO: Hello SoftAP from loop()!
configured credentials:
ssid=myrouter security=wpa2 cipher=1
available access points:
SSID=Habs security=wpa2 channel=6 rssi=-78
SSID=myrouter security=wpa2 channel=11 rssi=-71
SSID=myrouter-guest security=wpa2 channel=11 rssi=-71
SSID=randomwifi1 security=unsecured channel=11 rssi=-70
SSID=DIRECT-A6-HP OfficeJet Pro 8710 security=wpa2 channel=11 rssi=-79
0000040765 [app] INFO: Hello SoftAP from loop()!
0000045766 [app] INFO: Hello SoftAP from loop()!

Doing this experiment, I observe that once the wifi credentials are set via softAP, the photon shortly breaths blue, then blinks green, then finally breaths green and stays there.
While breathing green, I cannot “signal” the photon in this case, so here the photon is REALLY not connected to the cloud.
If I reset it, it goes into breathing blue (which I think is the normal state for this wifi debug firmware).

Question

What can be going on inside the Photon’s mind?

Please reply if you have any hints.
Thanks!

DeviceOS

I have tested this with:

  • 2.0.1 works!
  • 2.1.0 works!
  • 2.2.0-rc.1 works!
  • 2.2.0-rc.2 does NOT work
  • 2.2.0: does NOT work
  • 2.3.0: does NOT work
  • 3.2.0: does NOT work

EDIT: I have tested different deviceOS versions and it seems like this issue got introduced in 2.2.0-rc2.

DeviceOs 2.2.0-rc2 release notes:

ENHANCEMENTS

  • Refactor system describe/info JSON generation to reduce size and remove invalid modules from it #2347 #2349

BUGFIXES

  • Clear module slots in DCT when preparing for an OTA update #2346
  • Do not reset the DTLS session on socket errors #2335 #2337
  • [Argon / Tracker] Avoid power leakage through ESP32 ESPBOOT pin #2342
  • Fix parsing of JSON strings with more than 127 tokens #2348

INTERNAL

  • [CI] Generate public Codefresh URL in Slack notifications #2333
  • [Photon / P1] system part 2 size optimizations #2349
  • [Electron] Increase MBEDTLS_SSL_MAX_CONTENT_LEN to 900 #2349

I scratched few out.
Do you want to bet on where could the fault be?

@Colleen can you help me route this issue to the appropiate person? Thanks!

1 Like
2

Thanks for pointing this out to us @gusgonnet. The great @lanceseidman will work on taking care of this!

2 Likes
3

Hello,

We’re working with the Engineering team to help identify this issue, please give us some time to investigate this further.

3 Likes
4

Hey @lanceseidman any progress here? I know y’all are working on it, just trying to figure out if we should postpone the wifi version of our product or if there’s a fix coming soon, go ahead and roll it out.

5

Hey, one valid mitigation strategy here could be to go back to DeviceOS 2.1.0.
Cheers!

1 Like
6

Hey Lance, any news on this front?
thanks

1 Like
7

Hello,

Yes, it was able to be reproduced.

“I did see the issue where after you enter credentials it tries to connect and falls back to blinking blue. I simulated this by jamming my finger into the chip antenna to attenuate it and prevent it from connecting. I suppose the same thing will happen if you use the wrong password. This is normal, and happens after a 21 second timeout of not being able to connect. After about 40 more seconds of being back in Listening Mode, it will try to connect again and this time it stays blinking green.”

So at this time, you can write firmware to test for connection timeouts and force the device back into Listening Mode if you want to signal better to the customer/yourself that it’s not connected and doesn’t have credentials.

8

@lanceseidman what they reproduce is not the issue. In my case, the credentials are valid, and the photon does not connect to the cloud.

Why? I did not do this.

Should it not connect to the cloud and breath cyan?

It does not seem that my firmware is running.

I'm very sad that I did a lot of tests, I was able to pinpoint at what deviceOS this issue starts happening, wrote in length how to reproduce and still I was not able to communicate this issue better.
I will try better next time,
Gustavo.

3 Likes
9

@lanceseidman any further input on this?

10

This topic was automatically closed 60 days after the last reply. New replies are no longer allowed.