I hope you're all doing well. I just wanted to drop in and confirm that we're fully committed to resolving the current issues and I want to assure you that the Realtek chipset itself is not the cause. We are however actively investigating some unusual system behaviors in devices that use this IC purely from a configuration point of view.
The 5.6 release branch has successfully addressed a number of latency problems in the ethernet, but we acknowledge that the reachability issue remains a separate challenge. Rest assured, our team is diligently working on it and I'll swing back later after we've discussed in more detail.
A polite ask here - when is OS 5.6.0 likely to land? Thanks
The background is my customer has bought 4000 P2s on the promise from Particle that Ethernet and WPA Enterprise would work. They are getting very concerned about the delays in having these features supported.
I appreciate the input from @mrlambchop - that was 2 weeks ago - any update?
Installed 5.6.0 toolchain and tried the same sketch for Photon2 that wasn't working with OS 5.5.0 - and it connects (eventually) via ethernet but no where near the smooth and quick connection experienced with Gen3. I will post the log here to see if there is an obvious reason - will repeat a few times first to see if this is repeatable.
As expected the Ethernet LAN connection time is quick however the cloud connection time is never less than 60 seconds and appears to contain a lot of errors
0000039371 [comm.coap] TRACE: CON GET /e/particle?u size=17 token= id=1232
0000039745 [comm.coap] TRACE: Retransmitting CoAP message; ID: 1231; attempt 1 of 3
0000040105 [net.en] ERROR: Failed to allocate pbuf
0000040121 [comm.coap] TRACE: Sending CoAP message
0000040276 [net.en] ERROR: Failed to allocate pbuf
0000040290 [comm.coap] TRACE: CON POST /d?\x02 size=25 token=3f id=1231
0000040438 [net.en] ERROR: Failed to allocate pbuf
0000040518 [comm.coap] TRACE: Retransmitting CoAP message; ID: 1230; attempt 1 of 3
0000040690 [net.en] ERROR: Failed to allocate pbuf
0000040709 [comm.coap] TRACE: Sending CoAP message
0000040864 [net.en] ERROR: Failed to allocate pbuf
0000040879 [comm.coap] TRACE: CON GET /t size=7 token=3e id=1230
0000041053 [net.en] ERROR: Failed to allocate pbuf
0000041152 [comm.coap] TRACE: Received CoAP message
0000041632 [comm.coap] TRACE: CON GET /d?\x02 size=9 token=01 id=53418
0000041735 [net.en] ERROR: Failed to allocate pbuf
0000041754 [comm.coap] TRACE: Retransmitting CoAP message; ID: 1228; attempt 2 of 3
0000041909 [net.en] ERROR: Failed to allocate pbuf
0000041927 [comm.coap] TRACE: Sending CoAP message
0000042075 [net.en] ERROR: Failed to allocate pbuf
0000042089 [comm.coap] TRACE: CON POST /E/particle/device/updates/enabled size=44 token= id=1228
0000042238 [net.en] ERROR: Failed to allocate pbuf
0000042321 [comm.protocol] INFO: Received DESCRIBE request; flags: 0x02
0000042500 [net.en] ERROR: Failed to allocate pbuf
0000042529 [comm.coap] TRACE: Sending CoAP message
0000042665 [net.en] ERROR: Failed to allocate pbuf
0000042680 [comm.coap] TRACE: ACK 0.00 size=4 token= id=53418
0000042828 [net.en] ERROR: Failed to allocate pbuf
0000042912 [comm.coap] TRACE: Sending CoAP message
0000043110 [net.en] ERROR: Failed to allocate pbuf
0000043137 [comm.coap] TRACE: CON 2.05 size=21 token=01 id=1234
0000043272 [net.en] ERROR: Failed to allocate pbuf
0000043371 [comm.coap] TRACE: Received CoAP message
0000043723 [comm.coap] TRACE: ACK 0.00 size=4 token= id=1228
0000043958 [net.en] ERROR: Failed to allocate pbuf
0000043982 [comm.coap] TRACE: Retransmitting CoAP message; ID: 1229; attempt 2 of 3
0000044130 [net.en] ERROR: Failed to allocate pbuf
0000044146 [comm.coap] TRACE: Sending CoAP message
0000044295 [net.en] ERROR: Failed to allocate pbuf
0000044320 [comm.coap] TRACE: CON POST /E/particle/device/updates/forced size=44 token= id=1229
0000044456 [net.en] ERROR: Failed to allocate pbuf
0000044556 [comm.coap] TRACE: Received CoAP message
0000044637 [comm.coap] TRACE: ACK 0.00 size=4 token= id=1229
0000045088 [comm.coap] TRACE: Received CoAP message
0000045467 [net.en] ERROR: Failed to allocate pbuf
0000045493 [comm.coap] TRACE: CON POST /E/particle/device/updates/pending size=47 token=02 id=53419
0000045636 [net.en] ERROR: Failed to allocate pbuf
0000045666 [comm.coap] TRACE: Sending CoAP message
0000045802 [net.en] ERROR: Failed to allocate pbuf
0000045829 [comm.coap] TRACE: ACK 0.00 size=4 token= id=53419
0000045964 [net.en] ERROR: Failed to allocate pbuf
0000046062 [comm.coap] TRACE: Received CoAP message
0000046364 [net.en] ERROR: Failed to allocate pbuf
0000046376 [comm.coap] TRACE: ACK 0.00 size=5 token=3f id=1231
0000046524 [net.en] ERROR: Failed to allocate pbuf
0000046540 [comm.coap] TRACE: Retransmitting CoAP message; ID: 1233; attempt 2 of 3
0000046689 [net.en] ERROR: Failed to allocate pbuf
0000046706 [comm.coap] TRACE: Sending CoAP message
0000046860 [net.en] ERROR: Failed to allocate pbuf
0000046875 [comm.coap] TRACE: CON GET /e/spark?u size=14 token= id=1233
0000047023 [net.en] ERROR: Failed to allocate pbuf
0000047090 [comm.coap] TRACE: Retransmitting CoAP message; ID: 1232; attempt 2 of 3
0000047275 [net.en] ERROR: Failed to allocate pbuf
0000047292 [comm.coap] TRACE: Sending CoAP message
0000047447 [net.en] ERROR: Failed to allocate pbuf
0000047462 [comm.coap] TRACE: CON GET /e/particle?u size=17 token= id=1232
0000047610 [net.en] ERROR: Failed to allocate pbuf
0000047691 [comm.protocol] TRACE: Updating application DESCRIBE checksum
0000047864 [net.en] ERROR: Failed to allocate pbuf
0000047880 [comm.dtls] INFO: session cmd (CLS,DIS,MOV,LOD,SAV): 4
0000048034 [net.en] ERROR: Failed to allocate pbuf
0000048050 [comm.dtls] INFO: session cmd (CLS,DIS,MOV,LOD,SAV): 3
0000048198 [net.en] ERROR: Failed to allocate pbuf
0000048361 [net.en] ERROR: Failed to allocate pbuf
0000048551 [net.en] ERROR: Failed to allocate pbuf
0000048617 [comm.coap] TRACE: Received CoAP message
0000049079 [comm.coap] TRACE: ACK 0.00 size=4 token= id=1232
0000049468 [net.en] ERROR: Failed to allocate pbuf
0000049482 [comm.coap] TRACE: Retransmitting CoAP message; ID: 1234; attempt 1 of 3
0000049637 [net.en] ERROR: Failed to allocate pbuf
0000049654 [comm.coap] TRACE: Sending CoAP message
0000049802 [net.en] ERROR: Failed to allocate pbuf
0000049816 [comm.coap] TRACE: CON 2.05 size=21 token=01 id=1234
0000049964 [net.en] ERROR: Failed to allocate pbuf
0000050037 [comm.coap] TRACE: Retransmitting CoAP message; ID: 1230; attempt 2 of 3
0000050221 [net.en] ERROR: Failed to allocate pbuf
0000050239 [comm.coap] TRACE: Sending CoAP message
0000050387 [net.en] ERROR: Failed to allocate pbuf
0000050401 [comm.coap] TRACE: CON GET /t size=7 token=3e id=1230
0000050550 [net.en] ERROR: Failed to allocate pbuf
0000050649 [comm.coap] TRACE: Received CoAP message
0000050951 [net.en] ERROR: Failed to allocate pbuf
0000050971 [comm.coap] TRACE: CON GET /d?\x02 size=9 token=01 id=53418
0000051113 [net.en] ERROR: Failed to allocate pbuf
0000051141 [comm.coap] TRACE: Sending CoAP message
0000051277 [net.en] ERROR: Failed to allocate pbuf
0000051290 [comm.coap] TRACE: ACK 0.00 size=4 token= id=53418
0000051445 [net.en] ERROR: Failed to allocate pbuf
0000051544 [comm.coap] TRACE: Received CoAP message
0000051882 [comm.coap] TRACE: ACK 0.00 size=4 token= id=1233
0000052459 [comm.protocol] TRACE: Updating subscriptions checksum
0000052619 [net.en] ERROR: Failed to allocate pbuf
0000052637 [comm.dtls] INFO: session cmd (CLS,DIS,MOV,LOD,SAV): 4
0000052792 [net.en] ERROR: Failed to allocate pbuf
0000052808 [comm.dtls] INFO: session cmd (CLS,DIS,MOV,LOD,SAV): 3
0000052956 [net.en] ERROR: Failed to allocate pbuf
0000053013 [comm.coap] TRACE: Received CoAP message
0000053264 [net.en] ERROR: Failed to allocate pbuf
0000053277 [comm.coap] TRACE: CON POST /E/particle/device/updates/pending size=47 token=02 id=53419
0000053432 [net.en] ERROR: Failed to allocate pbuf
0000053453 [comm.coap] TRACE: Sending CoAP message
0000053601 [net.en] ERROR: Failed to allocate pbuf
0000053615 [comm.coap] TRACE: ACK 0.00 size=4 token= id=53419
0000053763 [net.en] ERROR: Failed to allocate pbuf
0000053862 [comm.coap] TRACE: Received CoAP message
0000054246 [comm.coap] TRACE: ACK 0.00 size=4 token= id=1229
0000055007 [comm.coap] TRACE: Received CoAP message
0000055274 [comm.coap] TRACE: ACK 0.00 size=4 token= id=1228
0000055619 [comm.coap] TRACE: Received CoAP message
0000055911 [comm.coap] TRACE: ACK 0.00 size=4 token= id=1227
0000056182 [comm.coap] TRACE: Received CoAP message
0000056296 [comm.coap] TRACE: CON GET /d?\x02 size=9 token=01 id=53418
0000056692 [comm.coap] TRACE: Sending CoAP message
0000056868 [comm.coap] TRACE: ACK 0.00 size=4 token= id=53418
0000057280 [comm.coap] TRACE: Received CoAP message
0000057421 [comm.coap] TRACE: ACK 0.00 size=4 token= id=1233
0000057688 [comm.coap] TRACE: Received CoAP message
0000057803 [comm.coap] TRACE: CON POST /E/particle/device/updates/pending size=47 token=02 id=53419
0000058022 [comm.coap] TRACE: Sending CoAP message
0000058406 [comm.coap] TRACE: ACK 0.00 size=4 token= id=53419
0000058797 [comm.coap] TRACE: Received CoAP message
0000058936 [comm.coap] TRACE: ACK 0.00 size=4 token= id=1232
0000059330 [comm.coap] TRACE: Received CoAP message
0000059465 [comm.coap] TRACE: ACK 0.00 size=5 token=3f id=1231
0000059745 [comm.coap] TRACE: Received CoAP message
0000059854 [comm.coap] TRACE: ACK 2.05 size=10 token=3e id=1230
0000059992 [comm.coap] TRACE: Retransmitting CoAP message; ID: 1234; attempt 2 of 3
0000060210 [comm.coap] TRACE: Sending CoAP message
0000060354 [comm.coap] TRACE: CON 2.05 size=21 token=01 id=1234
0000060798 [comm.protocol] INFO: Received TIME response: 1702141956
0000061289 [comm.coap] TRACE: Received CoAP message
0000061411 [comm.coap] TRACE: ACK 0.00 size=4 token= id=1228
0000061773 [comm.coap] TRACE: Received CoAP message
0000062135 [comm.coap] TRACE: ACK 0.00 size=4 token= id=1234
0000062308 [system] INFO: All handshake messages have been processed
0000062472 [comm.protocol] TRACE: Updating application DESCRIBE checksum
0000062633 [comm.dtls] INFO: session cmd (CLS,DIS,MOV,LOD,SAV): 4
0000062774 [comm.dtls] INFO: session cmd (CLS,DIS,MOV,LOD,SAV): 3
0000063140 [comm.coap] TRACE: Received CoAP message
0000063562 [comm.coap] TRACE: ACK 0.00 size=4 token= id=1229
0000063741 [system] INFO: Cloud connected
0000063833 [comm.coap] TRACE: Received CoAP message
0000063838 [comm.coap] TRACE: ACK 0.00 size=4 token= id=1233
0000063833 [app.ethwifi] INFO: Cloud connected over Ethernet keepAlive=25
0000063851 [comm.coap] TRACE: Received CoAP message
0000063952 [comm.coap] TRACE: ACK 0.00 size=4 token= id=1232
0000064168 [comm.coap] TRACE: Received CoAP message
still connects to on the ethernet work provided by my mac (internet sharing). it is still slow (20s to 30s).
still does not connect to my actual lan. lots of coap stuff but never connects (timeout set to 10 minutes). sometimes dns does not even work (any other device, including the argon, has no problem on this network)
@marekparticle@rickkas7@avtolstoy@no1089 given all the logs and pcaps we sent: you must have some idea what is going on? can you give us some insight? is there a telltale sign on networks that do not work?
we cannot afford customer calls, certainly not when the issue is this vague (customers expect ethernet to just work)
it is actually different. it will not connect (directly connected to unifi gateway), not even if you give it all the time in the world
sometimes i see it cannot resolve the address <-- id here -->.v5.udp.particle.io#5684
sometimes it gets stuck at the coap stuff (e.g. Retransmitting CoAP message; ID: 8; attempt 3 of 3)
i do not see any log lines at level "error" and no mention about pbuffer
as for reproducing: i tried many things. adding switch/router etc to the ethernetwork provided by my mac (still works) as well as changing the actual lan. i upgraded the gateway to a ubiquiti unifi dream machine se and have a new isp bridge. (still does not work)
last thing i will do is to setup my old gateway again, make sure i get the problem and then ship that to you
What I have done is change my router since the OS 5.5.0 test. I now have the same internet modem but this then goes into a TPLink Deco WiFi6 Mesh and the ethernet connection is to one of the mesh nodes.
I noticed that it didn't work (connect to ethernet) initially then once through that subsequent connections (ethernet and cloud) were quicker.
I just retried it - ethernet connected in 3 seconds and cloud in 13 seconds. Once connected it is very stable.
Frank - sorry to hear that this OS release isn't any improvement for you. I completely agree with customer expectations for ethernet - it should be plug and on/working.
A reliable ethernet connection solution is one thing we will require before we ship because this must overcome the WiFi problems we face with Photon1. The other thing we will need is WPA Enterprise (eduroam) - I am still waiting on an answer about that from @Colleen ?
Still taking an active interest in this, as this does have implications for how we will need to reduce our risk as we start our hardware spin in the new year.
Wondering if there has been any development with this. We have been experiencing similar issues when trying to use the Ethernet connection for the cloud connection. When it does connect it is very slow, drops out frequently, and the local ping time steadily increases to an unusable degree.
@boddeke and channel - 5.7 is out and should fix these remaining edge cases, please test and let us know! should be out on workbench later today or tomorrow.
