WiFi reconnection issue - what does this trace mean?

[UMD]

Currently in the middle of investigating a long standing issue that others have found:

Once a WiFi connection is lost, it can only be re-instigated by a reset.

This is bad.

I note that this issue was seemingly solved here: [SOLVED] WiFi Reconnection Issue - Is there a solution?) based on @rickkas7 WiFi State Machine technique.

So, why has it not remained solved for me?

THEORY is that it has something to do with the SPI interface. Am now using the SDFAT 0.7 library, but before was not… More to follow on this as I progress

In the mean time, to assist with the investigation, can anyone respond to what causes the following [hal] and [hal.wlan] traces?

====== ETH cable pulled ==============
0000379927 [hal.wlan] TRACE: connect cancel
0000379928 [hal] TRACE: 20015b1c socket list: 0 active sockets closed
0000379928 [hal] TRACE: 20015b24 socket list: 0 active sockets closed
0000390242 [app] WARN: WiFi.ready()=0 WiFi.localIP()=0  

Thanks!

[armor]

Which device OS are you using that exhibits this behaviour?

What do you mean by WiFi Connection is lost - AP turned off or out of range or internet connection lost?

It would appear that you have an Argon in an Ethernet Featherwing is that the case? And you have unplugged the cable?

[UMD]

@armor, I should have been more specific!

DeviceOS 1.5.0 on a P1.

The issue is when a connection drops for whatever reason.

I am simulating by removing the ethernet cable from the access point (ie loss of internet) and also turning off the access point (loss of WiFi), then re-instating. Problem for me is that the cloud connection does not come back, even though the access point is operational.

Am now circling my coding being at issue (isn’t that the case in 90+% of the time!) because I resurrected this solution which works with the tests above:

[SOLVED] TCPCLIENT intranet connection fails if no cloud connection

This example should be closer to what you want: #include "Particle.h" SYSTEM_THREAD(ENABLED); SYSTEM_MODE(SEMI_AUTOMATIC); const unsigned long SEND_PERIOD_MS = 15000; enum WifiState { WIFI_STATE_NOT_CONNECTED, WIFI_STATE_CONNECTING, WIFI_STATE_CONNECTED, WIFI_STATE_RUNNING }; IPAddress serverAddr(192,168,2,186); const int serverPort = 8081; WifiState wifiState = WIFI_STATE_NOT_CONNECTED; unsigned long lastSend = 0; TCPClient client; int seq = 0; void setup() { Serial.begin(9600); WiFi.…

I will report back once I have more to say. Any insights are always appreciated!

[armor]

I tried devices this last weekend (they are on Photon and have SPI to a TFT and SD card) on 1.5.0 and I had a number of issues related to SPI so have reverted to 1.4.4 and will sit out 1.5.0 until it is clear the SPI and connection issue have been solved or at least it is clear what has changed to invalidate the program models that were working and how they need to be remediated.

[UMD]

@armor, I did say that the SPI / SD CARD was a theory in relation to my issue, but too early to be definitive. I note that my issue pre-dates DeviceOS 1.5.0.

Wrt your SPI issues, refer to Updating from DeviceOS 1.4.4 to 1.5.0-rc.2 -> Broken SPI Functionality. There was an issue of Display vs SD CARD in the release candidates, but with DeviceOS 1.5.0 I have no had an issue with issue with these two peripherals, so not sure what is going on. Suggest that you raise a separate topic for this.

[UMD]

Happy to report that I have found the reason for the issue:

Once a WiFi connection is lost, it can only be re-instigated by a reset.

It was lack of enough free memory.

It had nothing to do with SPI or erroneous code (but solved by code of course by reducing memory usage).

This also explains why what was once working now was not.

[Elco]

I think the interaction with SPI is that the SPI class uses a recursive os mutex. If there is no free memory, allocation fails, the mutex cannot get a lock and a deadlock occurs.

I have seen it before when hardware debugging. The code hangs on a mutex in the app thread, but the real issue is running out of memory, which makes the mutex deadlock.

I have removed any use of the SPI class now and use raw HAL functions instead. If you know how you use SPI, you don’t need all these mutexes in between on every SPI call.

The particle classes are very defensive and oriented at beginners, so in many places in my application I have just looked up what they do and implemented the same using HAL functions without all the bloat.

I prefer managing my own mutex using std::mutex and std::unique_lock<std::mutex> if I need one, to manage ownership and automatic unlocking on destruction.

[UMD]

@Elco, now that is very interesting indeed! Makes sense.

Have you raised a support ticket for your finding as per @ScruffR comments here: WiFi.ready() == FALSE but WiFi is connected ?

@avtolstoy, any comment on @Elco’s statement:

I think the interaction with SPI is that the SPI class uses a recursive os mutex. If there is no free memory, allocation fails, the mutex cannot get a lock and a deadlock occurs.

Re accessing the HAL functions directly, how is this done? I did not think that this could be done from the application. If so, great!

[Elco]

Yes, I also sent in a ticket and got a reply.

Well, I don’t know how you develop, but I use vscode and makefiles. So I have the entire particle device-os repo as a dependency in the same directory. So I just browse the device-os files to see how they implemented the SPI class (“spark_wiring_spi.h” / “spark_wiring_spi.cpp”) and re-used code from it.

But this is only for experienced embedded software developers that can fully understand what the particle code is doing. This is not documented and particle doesn’t expect you to use it this way.

The documentation often leaves out crucial details, like the exact function prototypes with the type of arguments and possible overloads. So I end up browsing the code anyway.

I think my next hardware revision will not use a Particle device anymore, but a bare ESP32 instead. I don’t use their cloud and the easy to use framework often gets in my way more than it helps me. I’m just not their target market. I’m an early adopter that got in during the Spark Core days, when ESP32’s were not even a thing. I have the skills to develop on bare metal and don’t need Arduino style libs.

[UMD]

@Elco, I now understand how you are linking to the HAL layers - you are compiling the whole code base ie DeviceOS and Application, hence have access to all the DeviceOS functions.

I use Particle Dev as I don’t want the overhead of maintaining the environment, so this avenue is not available to me.

I understand your frustration…

[Elco]

I still use the system layers, not a monolithic build. But yes, I go deep into the framework.
Our code is also built in the cloud by Azure and automatically tested and deployed, so that’s why we use makefiles too.

[ScruffR]

@UMD

I use Particle Dev as I don’t want the overhead of maintaining the environment, so this avenue is not available to me.

Can you elaborate what kind of maintance effort you are anticipating that keeps you from transitioning to Workbench?

[UMD]

@ScruffR, good question. The inertia is there due to convenience (you know, devil you know vs the devil you don’t).

I am by no means versed in the using repositories, and so have been assuming that there was an overhead in keeping in sync with the latest DeviceOS incarnations.

From your question I take it that Workbench is the preferred option for serious development. I shall give it a go in a few months.

[UMD]

@no1089,

Thought it appropriate that I respond to the communications that we had under the now closed post:

Long Term Photon Connection Stability

Am using DeviceOS 1.5.2 and suffering the same “long term” issue of flashing cyan and not recovering connectivity. Is there a definitive reason for this ongoing and frustrating issue? Dare I say this is my last problem to solve before I am comfortable going into production… The number of contributors and contributions to this post is a testament to how bad this issue is. IMHO it is really poor that after all these years that it has not been sorted by official Particle advice, and/or DeviceOS…

The issue that I was complaining about in the above post, and this post:

[SOLVED] WiFi Reconnection Issue - Is there a solution?

Problem in short is that once a WiFi connection is dropped, eg by disabling and renabling your WiFi hotspot, the Photon does not reconnect again until one reboots the Photon. This is a real problem in the field! This is very similar, if not one and the same, as the following issues: @mhazley as described here [v0.4.8-rc1 / v0.4.9] WiFi Reconnection Issue and here Failures on handling WiFi reconnects ( v0.4.5 / v0.4.6 ) @ekarlso here Reconnection wifi issues 0.4.9 @rebelzach here Photon loses…

has been (re) solved today.

In short:

• Implement a WiFi connection state machine
• Ensure that there is enough heap memory

It seems that DeviceOS 1.5.2 has larger memory requirement and that is what unexpectedly threw up my recent woes (hence the frustration).

I believe that Particle are working on reducing DeviceOS memory requirements and if this can be improved, that would be great moving forward.

Thanks!

[no1089]

I have seen the same issue you noted, a Photon not reconnecting to an AP if the AP is power cycled, and I agree, it’s frustrating to deal with.
I will check internally if this is specifically being worked on.

Yes, the 2.0 LTS release thread memory reduction will hopefully address many issues.

From your post, you are unblocked for the moment?

[UMD]

@no1089, confirming that am all good.

The implemented “WiFi state machine” works a treat when I don’t run too low on memory.

[dan.s]

Thanks for your results @UMD!
We have come across this problem recently, so I’m looking at implementing the WiFI state machine (from [SOLVED] TCPCLIENT intranet connection fails if no cloud connection right?) and I wondered if you came to a rough figure of the available heap memory to keep free?

[UMD]

@dan.s, good question re memory.

Memory is especially an issue with later versions of DeviceOS.

Here are some posts for you to ponder:

Photon/P1 Unable to connect to setup WiFi after upgrade to OS v1.5.0

With the same firmware code. After I have upgraded from OS v1.4.4 to v1.5.0, I am not able to setup WiFi through serial or through the softAP anymore. I did discover during listening mode (softAP on) I had around 17000 Bytes of free memory with my firmware in OS v1.4.4 and went to around 1000 Bytes of free memory with the same firmware in OS v1.5.0. With SoftAP off, 35000 bytes for v1.4.4 and 25000 bytes for v1.5.0 I freed modified my firmware to have around 5000 Bytes of free memory without su…

Device OS 1.5.2 Memory Use

I was doing some regression testing of an application built on 1.5.2 - the wifi setup using SoftAP didn’t work because the memory was too low. Checking the vitals I noticed that 50kB was in use out of 81kB RAM available. I tried rebuilding the same application on 1.4.4 and the vitals now show 39kB in use. That’s a big jump in OS RAM usage - have I missed something in the release notes about this?

etc…

Mention was made here that 10K of free memory was required when using SoftAP:

Using Soft AP example - can't connect to HTTP

UPDATE: I was able to get reliable ping AND connection to the HTTP service on port 80 with the device. How? I put the device into SAFE MODE then into LISTEN MODE… I believe this to be the default SoftAP mode of operation and not my code, because there was no head or body content, just the the HTML tags.

My guess - at least 10K of free memory is required. You need to instrument your code using freeMemory() to test.

[ScruffR]

IIRC you shouldn’t use SoftAP with less than 20KB free memory (before first time entry into Listening Mode).

[dan.s]

Yes the softAP memory issue is a particular thorn for us, you can see my easy fix at Listening mode on the Photon cannot work reliably in current implementation

Is this reconnection issue related to softAP?