Thanks, @MORA. I at least want to pass this on to our board designer. I feel like something else is going on too. I’ve noticed going straight to listening mode seems to never work, but if I clear the stored credentials, I can hit reset and stay in listening mode, which works most of the time. Sometimes it takes more hard resets and/or power cycles though. Really weird.
Okay, managed to build my application locally, program-dfu didnt work with it, but I flashed the bin with “particle flash” so that should ensure it was compiled locally.
But the issue it the same, “particle serial monitor” stops outputting text when I go to listening mode 
Spark Core or Photon says Hello over USB! 61
Spark Core or Photon says Hello over USB! 62
0000066665:DEBUG: int Spark_Disconnect() (634):
0000066670:DEBUG: int Spark_Disconnect() (643):Close Attempt
0000066677:DEBUG: sock_result_t socket_close(sock_handle_t) (936):socket closed
20005e68
0000066686:DEBUG: int Spark_Disconnect() (645):socket_close()=0
0000066692:INFO : void cloud_disconnect(bool) (410):Cloud: disconnectedI noticed the same, I even made the mode button clear credentials to test it.
With cleared credentials and thereby forced listneing mode I have around 1 in 5 attempts work, with my app running and credentials stored before going to listen its more like 1 in 20.
And finally with a simple user app or tinker loaded its around the same as with forced listening mode.
But debugging the issue is pretty hard since theres no solid number to look at to see if it gets better…
Your application debug will stop when entering listening mode, since the application loop doesn’t run when in listeningmode. The system debug output. (assuming you are seeing system debug statements?) will continue.
0000066692:INFO : void cloud_disconnect(bool) (410):Cloud: disconnected
Is that not a system debug statement ?
Its the last line I get before going to listening mode.
Yep, that’s a system debug statement. Did you add debug statements to the softap.cpp file? It’s a good idea to add lots of them so you see where the system is executing code related to softap.
To start with I’d log the result of softAP.start(); in SoftAPApplication constructor.
I added 5 debug statements to the softAP start call, but even without those, I would expect some debug output from all the other code being run ?
wiced_result_t start() {
LOG("SoftAP start called");
wiced_result_t result;
if (!(result=wiced_rtos_init_semaphore(&complete)))
{
LOG("IF1");
if (!(result=setup_soft_ap_credentials()))
{
LOG("IF2");
if (!(result=wiced_network_up( WICED_AP_INTERFACE, WICED_USE_INTERNAL_DHCP_SERVER, &device_init_ip_settings )))
{
LOG("IF3");
if (!(result=wiced_dns_redirector_start( &dns_redirector, WICED_AP_INTERFACE )))
{
LOG("IF4");
result = WICED_SUCCESS;
}
}
}
}
LOG("Result %d", result);
return result;
}I’m not sure what debug statements there are in listening mode, so yours may be the only ones output.
I suggest switching to MANUAL mode, and only initializing the WiFi connection (WiFi.on(), WiFi.listen()) to reduce the amount of debug logs.
Ok, I will try that tonight, I have been using the setup button to enter listening mode, rather than wifi.listen, if that makes any difference.
good things first, I got debugging working 
C:\Users\MORA\particle\firmware\modules>particle serial monitor
Opening serial monitor for com port: "COM2"
0000002101:INFO : void manage_cloud_connection(bool) (289):Cloud: disconnecting
Spark Core or Photon says Hello over USB! 1
0000003113:INFO : void manage_cloud_connection(bool) (289):Cloud: disconnecting
Spark Core or Photon says Hello over USB! 2
0000004126:INFO : void manage_cloud_connection(bool) (289):Cloud: disconnecting
Spark Core or Photon says Hello over USB! 3
0000005139:INFO : void manage_cloud_connection(bool) (289):Cloud: disconnecting
Spark Core or Photon says Hello over USB! 4
0000006152:INFO : void manage_cloud_connection(bool) (289):Cloud: disconnecting
Spark Core or Photon says Hello over USB! 5
0000007165:INFO : void manage_cloud_connection(bool) (289):Cloud: disconnecting
Spark Core or Photon says Hello over USB! 6
0000008177:INFO : void manage_cloud_connection(bool) (289):Cloud: disconnecting
Spark Core or Photon says Hello over USB! 7
0000009190:INFO : void manage_cloud_connection(bool) (289):Cloud: disconnecting
Spark Core or Photon says Hello over USB! 8
0000010203:INFO : void manage_cloud_connection(bool) (289):Cloud: disconnecting
Spark Core or Photon says Hello over USB! 9
0000011636:LOG : wiced_result_t SoftAPController::start() (846):SoftAP start ca
lled
0000012645:LOG : wiced_result_t SoftAPController::start() (851):IF1 0
0000013652:LOG : wiced_result_t SoftAPController::start() (855):IF2 0
0000014797:LOG : wiced_result_t SoftAPController::start() (859):IF3 0
0000015804:LOG : wiced_result_t SoftAPController::start() (863):IF4 0
0000017811:LOG : wiced_result_t SoftAPController::start() (871):Result 0
0000017818:LOG : wiced_result_t SoftAPController::start() (872):Wiced success 0
But as the log shows all of those calls return WICED_SUCCESS (0).
Bad news is it seems the trail ends at “wiced” calls which we dont have the source for.
So I dont really know whats behind “wiced_network_up( WICED_AP_INTERFACE, WICED_USE_INTERNAL_DHCP_SERVER, &device_init_ip_settings )”
Thanks for your perseverance so far That’s a great piece of knowledge to have. 
I do have access to the WICED sources, but my guess is that as far as the device is concerned, it think it has enabled DHCP. The WICED code is quite thorough when it comes to error checking.
The common factor in all the reported cases is that it happens when the Photon is mounted to a custom PCB. So I wonder if all the right commands are being sent to the WiFi module but it is failing to execute them correctly.
Does your PCB have a JTAG port? If so, I’m happy to try to help debug if you can send me a PCB.
Niether of my PCBs have a JTAG port, but if all the pins required are on the side of the photon, one could be hacked onto it.
Yes, I can send you some test PCBs, in what state of assembledness do you want ?
I tested the debug code on my deadbug test with the board and it worked still, thats where the photon is soldered onto a empty board.
My test above was with usb power too, so the suspected regulator should be out of question, but maybe I am doing other bad things to the photon in my setup.
Maybe I should start soldering components onto the deadbug to see when it fails 
My schematic for the current board can be seen here : https://dl.dropboxusercontent.com/u/38342600/SBNG-Mnin.pdf
Soldering components until failure sounds like a very good plan. Please try that first then we can look at getting a PCB to me later.
We know it’s something on the board since the problem doesn’t happen with unmounted photons.
@MORA, your pdf link is not working.
Hm, dropbox mumble mumble …
I uploaded it here instead if anyone is interested : https://stokerlog.dk/exports/SBNG-Mnin.pdf
That didnt really help me get closer to the issue.
I soldered one component after another and tested that I could do the softap setup inbetween, I managed to get all non-mechanical components on the board without it making a issue.
I then removed the old capacitor on my test board (that I did the serial debug on) and replaced it with a new better quality one, and I could complete the wifi setup on it.
But as I wrote on github, initially when the device was running tinker I could complete the softap too, so I tried a few times and it worked.
I then tried to replace that capacitor on a 3rd board that wont accept softAP, and it didnt work, so went back and tried on the other two and they failed aswell, so the issue is still intermittent, where 2 out of 3 boards will work sometimes and the 3rd one rarely will.
By the way I tried to starve the photon of current on the bench supply to see what would happen, when limited to 200mA@5V it goes red in the RGB during softAP and reboots (Yes I realise its a bit risky with the low vin kills photon thread)
Could it be some sort of interferance? I am thinking if a car adapter and going to a empty area would by worthwhile, although your area in the big city is likely ten times more noisy than here.
After 2 hours I took the deadbug board out and tried softAP again which failed, then reflashed tinker via the mobile app and successfully performed softAP 2 times out of 2, then flashed my own app and tried a 3rd time which worked, so pretty hard to debug.
Maybe heat related, the 3rd board that wont play along runs 24/7, whereas the two others are not being used other than debugging.
Since this issue does not seem to be going away, I made a demo for an alternative setup that may be an acceptable alternative for others in the same situration.
When the photon has no credentials it will scan for an unsecured wifi connection and use that to get online, then one can submit new credentials to it from a cloud function.
This code will at least allow me to continue to develop the main app, I can convince beta testers to setup a open AP on their phone and use a web form to transmit new wifi credentials, maybe even technical endusers, although its not the most elegant solution.
I do hope particle finds the root cause of this though, theres at least 3 independent sources of this problem and “It only happens when mounted to a PCB” is not really helpful, despite good intentions.
SYSTEM_MODE(MANUAL);
#include "SparkJson/SparkJson.h"
/*
The setup routine only looks for open networks, if there is no credentials stored, so when moving to a new network, the user needs to delete old credentials...
https://docs.particle.io/guide/getting-started/modes/photon/
To erase the stored wifi networks on your Photon, hold the SETUP button for about ten seconds, until the RGB LED flashes blue rapidly.
You can also reset the Wi-Fi networks by holding the SETUP button and tapping RESET, then continuing to hold SETUP until the light on the Photon turns white. (This differs from the Core. Doing this action on the Core will result in a factory reset.)
*/
unsigned long old_time = millis();
uint8_t retry_count = 0;
int setupWiFi(String json)
{
Serial.printf("setupWifi called : %s",(char*)json.c_str());
StaticJsonBuffer<200> jsonBuffer;
JsonObject& root = jsonBuffer.parseObject((char*)json.c_str());
const char* ssid = root["ssid"];
int security = root["security"]; //0=Open 1=WEP 2=WPA 3=WPA2
const char* pass = root["password"];
int cipher = root["cipher"]; //0=Not hidden, 1=AES, 2=TKIP, 3=AES_TKIP
Serial.printf("Connect to SSID:%s PASS:%s SECURITY:%d CIPHER:%d", ssid, pass, security, cipher);
WiFi.clearCredentials();
Particle.process();
WiFi.setCredentials(ssid, pass,security,cipher);
Particle.process();
delay(1000);
System.reset();
///hal/inc/wlan_hal.h lines 120-150 defines the constants used, maybe converting the internal numbers to constants would be a good idea incase the value of the constants ever change...
}
void setup() {
Particle.function("setupWiFi", setupWiFi);
Serial.begin(9600);
delay(2000);
Serial.println("Alternative wifi setup demo...");
WiFi.on();
if (!WiFi.hasCredentials())
{
Serial.println("No credentials found, scanning for open wifis ...");
WiFiAccessPoint aps[20];
int found = WiFi.scan(aps, 20);
for (int i=0; i<found; i++) {
WiFiAccessPoint& ap = aps[i];
if (ap.security == 0 && strlen(ap.ssid) > 0)
{
Serial.printf("Found open AP %s", ap.ssid);
WiFi.setCredentials(ap.ssid);
Particle.process();
WiFi.connect();
retry_count=10;
while (retry_count-- > 0)
{
delay(1000);
if (WiFi.ready())
{
retry_count=0;
Serial.printf("Connected to %s", ap.ssid);
Serial.printf("Local IP is %s", String(WiFi.localIP()).c_str());
break;
}
}
if (WiFi.ready())
{
break;
} else {
WiFi.clearCredentials(); //This open network didnt work, try next one
}
}
}
if (!WiFi.ready())
{
Serial.println("Open network connection failed ...");
}
}
}
//Source kennethlimcp - https://github.com/kennethlimcp/particle-examples/blob/master/wifi-auto-reconnect/wifi-auto-reconnect.ino
void loop() {
if(millis() - old_time >= 2000){
if(retry_count < 10){
if(!WiFi.ready())
{
Serial.println("WiFi not ready, connect so stored credentials if any...");
WiFi.connect();
retry_count++;
}
else if (!Spark.connected()){
Serial.println("Connecting to cloud...");
Spark.connect();
retry_count++;
}
}
else{
Serial.println("Wifi settings didnt work after 10 tries ...");
WiFi.off();
retry_count = 0;
WiFi.on();
}
old_time = millis();
}
Particle.process();
// Insert user code here
}
Scope showing the vcc drop during listening mode, less than ideal, would that be enough for the particle to give up? despite a published min-vin of 3.6v ?
The drop is 380mV for about 60us before most is recovered.
Hey, @MORA. Based on some of the information in the GitHub discussion, we incorporated some changes in the latest revision of our board:
- Milled out the keep out area under the Photon
- Expanded the power feed from 40 mil to 60 mil
- Added a couple bigger decoupling capacitors in parallel to the Photon’s power input (0.1uF to 0.1uF, 1uF, and 10uF)
- Switched from a 500mA voltage regulator to a 1A regulator
- Added a 100uF capacitor to the existing 0.1uF, 1uF, and 1000uF capacitor’s on the power rail
I’m just on the software side myself. @Mahonroy is the hardware guy on this, so he can correct anything I may have said wrong. However, with a side by side comparison of the old and new boards, I did not seem to see the SoftAP issue anymore. I need to do some more testing with more boards and more iterations, but initially, it looked like I could just go to listening mode and connect to the AP without additional power cycles or messing around.
Thanks for your reply.
The issue has gotten a bit more attention over here Photon SMD Antena, keepout zones, and ground loops
In my 3rd revision I am using the header version of photon, and still having the keepout area, but with chinese new year it will be late february before I get my hands on that.
I may be able to get my board to a working state, since softAP works some times, so I guess the board is borderline ok, but every revision is a few weeks wait and while not expensive, not free either 