Photon freezing randomly (cyan stuck/not "breathing")

Hey everyone! I’ve got an issue that I can’t work out since it occurs randomly and without any interaction with the device.

I have a set of 8 Photon devices, each with an SD card reader (with SD card containing WAV files), button (with LED), strip of 20 neopixels, and speakers connected via PAM8610 amplifiers. Everything is running off a large 12V battery, providing 12V for the amplifiers, and running into a GB60-1205 DC/DC converter to provide 5V for the Photons. The Photons connect to a “pocket wifi” style mobile internet device.

Occasionally, one or more of the Photons will freeze and the LED will be stuck on cyan; an “offline” event appears in the console when this happens. This appears to be random and doesn’t require any interaction to have occurred first. I can’t think of what could be causing this, especially since it’s happening at random and not on all of the devices (so far at least).

Any ideas? (edit: I am isolating one of the culprits to see if it still crashes on a USB power supply)

Code is below. Apologies for the mess!

// This #include statement was automatically added by the Particle IDE.
#include <DeviceNameHelperRK.h>

// This #include statement was automatically added by the Particle IDE.
#include <FastLED.h>
#include <speaker.h>
#include <SdFat.h>
FASTLED_USING_NAMESPACE;

#define NUM_LEDS 20
#define LED_PIN 0
#define STRIP_PIN 6
#define BUT_PIN 1
#define UPDATES_PER_SECOND  60

String file_name [17]{ "1.wav","2.wav", "3.wav", "4.wav", "5.wav", "6.wav", "7.wav", "8.wav","9.wav",
                       "10.wav", "11.wav", "12.wav", "13.wav","14.wav","15.wav","16.wav","17.wav"};

String device_name = "none";

SerialLogHandler logHandler(LOG_LEVEL_NONE, {   // Logging level for non-app messages
    { "app", LOG_LEVEL_INFO }                   // Logging level for application messages
});

// Define the array of leds
CRGB leds[NUM_LEDS];
typedef struct wav_header {
    // RIFF Header
    char     riff_header[4];   // Contains "RIFF"
    uint32_t wav_size;         // Size of the wav portion of the file, which follows the first 8 bytes. File size - 8
    char     wave_header[4];   // Contains "WAVE"

    // Format Header
    char     fmt_header[4];    // Contains "fmt " (includes trailing space)
    uint32_t fmt_chunk_size;   // Should be 16 for PCM
    uint16_t audio_format;     // Should be 1 for PCM. 3 for IEEE Float
    uint16_t num_channels;
    uint32_t sample_rate;
    uint32_t byte_rate;        // Number of bytes per second. sample_rate * num_channels * Bytes Per Sample
    uint16_t sample_alignment; // num_channels * Bytes Per Sample
    uint16_t bit_depth;        // Number of bits per sample

    // Data
    char     data_header[4];   // Contains "data"
    uint32_t data_bytes;       // Number of bytes in data. Number of samples * num_channels * sample byte size
    // uint8_t bytes[];        // Remainder of wave file is bytes

} WavHeader_t;

const int    SD_SS      = A2;
const size_t BUFFERSIZE = 1024;
uint16_t     data[2][BUFFERSIZE];

SdFat        sd;
File         wavFile;
WavHeader_t  wh;

int buttonCount = 0;
int buttonState;
int lastButtonState = LOW;
unsigned long lastDebounceTime = 0;
unsigned long debounceDelay = 50;

static uint8_t hue;

Speaker speaker(data[0], data[1], BUFFERSIZE);

// handler id
int h = 0;

bool buttondown = false;

void setup() {
    delay( 3000 );
    Serial.begin(57600);
	  Serial.println("resetting");
    FastLED.addLeds<NEOPIXEL, STRIP_PIN>(leds, NUM_LEDS);  // GRB ordering is assumed
    //FastLED.setMaxPowerInVoltsAndMilliamps(5,10000); // limit my draw to 10A at 5v of power draw
    LEDS.setBrightness(100);
    pinMode(BUT_PIN, INPUT_PULLUP);
    pinMode(LED_PIN, OUTPUT);
    digitalWrite(LED_PIN, HIGH);
    Particle.function("playWav", selectFile);
    Particle.function("lights", ledChange);
    for(int i = 0; i < NUM_LEDS; i++) leds[i] = CHSV(hue,255,255);

    if (sd.begin(SD_SS)) Log.info("SD initialised");
    else Log.warn("failed to open card");

    Particle.subscribe("particle/device/name", subscriptionHandler);

}

void subscriptionHandler(const char *topic, const char *data){
    device_name = data;
}

//_______________________________________________________________________________
void loop() {

    if(device_name == "none"){
        Particle.publish("particle/device/name");
    }

    int reading = digitalRead(BUT_PIN);
    if (reading != lastButtonState) lastDebounceTime = millis();
    if ((millis() - lastDebounceTime) > debounceDelay) {
        if (reading != buttonState) {
            buttonState = reading;
            if (buttonState == HIGH) {
                buttonCount++;
    	          //String newSTR = buttonCount + ".wav";
                selectFile(file_name[buttonCount]); // REMOVE THIS TO DISABLE SOUND PLAYBACK ON BUTTON PUSH
                for(int i = 0; i < NUM_LEDS; i++) leds[i] = CHSV(hue++,255,255);
                Particle.publish("buttonpushed", device_name);
                buttondown = true;
            }
        }
    }
    lastButtonState = reading;
    if (buttondown && reading == LOW){
        buttondown = false;
        Particle.publish("buttonreleased", device_name);
    }
    if (buttonCount > 16) buttonCount = 0;
    FastLED.show();
    FastLED.delay(1000 / UPDATES_PER_SECOND);
    if (speaker.ready()) readChunk();
}

//_______________________________________________________________________________
int selectFile(const char* filename) {
    int retVal = 0;

    if (!strcmp("ls", filename) || !strcmp("dir", filename)) {
        sd.ls("/", LS_R);
        return 0;
    }

    if (wavFile.isOpen()) wavFile.close();

    wavFile = sd.open(filename);


    if (wavFile) {
        memset((uint8_t*)data, 0x80, sizeof(data)); // reset buffer to bias value 0x8080 (quicker via memset() than writing 0x8000 in a loop)
        if (sizeof(wh) == wavFile.read((uint8_t*)&wh, sizeof(wh))) {
        Log.printf("%s\n\r\t%.4s\r\n\tsize: %lu\r\n\t%.4s\r\n\t%.4s\r\n\tchunk size (16?): %lu\r\n\taudio format (1?): %u\r\n\tchannels: %u"
                , filename
                , wh.riff_header
                , wh.wav_size
                , wh.wave_header
                , wh.fmt_header
                , wh.fmt_chunk_size
                , wh.audio_format
                , wh.num_channels
                );
        // two chunks since Log only supports limited length output
        Log.printf("\r\n\tsample rate: %lu\r\n\tbyte rate: %lu\r\n\tsample alignment: %u\r\n\tbit depth: %u\r\n\t%.4s\r\n\tdata bytes: %u"
                , wh.sample_rate
                , wh.byte_rate
                , wh.sample_alignment
                , wh.bit_depth
                , wh.data_header
                , wh.data_bytes
                );
        retVal = wh.data_bytes;
        readChunk();
        speaker.begin(wh.sample_rate);
        }
    }
    else {
        Log.error("%s not found", filename);
    }

    return retVal;
}

int readChunk() {
    int retVal = 0;
    if (wavFile.isOpen()) {
        uint16_t* wav = speaker.getBuffer();
        uint8_t   buf[BUFFERSIZE * 2 * wh.num_channels];
        int       n = wavFile.read(buf, sizeof(buf));
        if (n < sizeof(buf)) {
            wavFile.close();               // when all data is read close the file
        }

        memset((uint8_t*)wav, 0x80, BUFFERSIZE);            // reset buffer to bias value 0x8080
        for(int b = 0; b < n; b += wh.sample_alignment) {
            wav[retVal++] = *(uint16_t*)&buf[b] + 32768;      // convert int16_t to uin16_t with bias 0x8000
        }
    } else {
        speaker.end();
    }
    return retVal;
}
//_______________________________________________________________________________

void fadeall() { for(int i = 0; i < NUM_LEDS; i++) { leds[i].nscale8(250); } }

//_______________________________________________________________________________

int ledChange(String preset){
    if(preset == "blue"){
        for(int i = 0; i < NUM_LEDS; i++) leds[i] = CRGB(0,0,255);
    } else if(preset =="red"){
        for(int i = 0; i < NUM_LEDS; i++) leds[i] = CRGB(255,0,0);
    } else if(preset =="green"){
        for(int i = 0; i < NUM_LEDS; i++) leds[i] = CRGB(0,255,0);
    } else if(preset =="yellow"){
        for(int i = 0; i < NUM_LEDS; i++) leds[i] = CRGB(255,255,0);
    } else if(preset =="pink"){
        for(int i = 0; i < NUM_LEDS; i++) leds[i] = CRGB(255,0,255);
    } else if(preset =="purple"){
        for(int i = 0; i < NUM_LEDS; i++) leds[i] = CRGB(127,0,255);
    } else if(preset =="cyan"){
        for(int i = 0; i < NUM_LEDS; i++) leds[i] = CRGB(0,255,255);
    } else if(preset =="off"){
        for(int i = 0; i < NUM_LEDS; i++) leds[i] = CRGB(0,0,0);
    } else {
        for(int i = 0; i < NUM_LEDS; i++) leds[i] = CHSV(hue++,255,255);
    }
    return 1;
}

With the speaker library plus using SPI for the SD card and FastLED you got a lot of time critical stuff going on - that combo increases the potential risk for deadlocks quite a bit.

There would be an easy to use HW alternative to the speaker library, that doesn’t break the bank either

Would it cause issues even if the SD card isn’t being read / no sound is playing? I’m getting crashes even before the SD card is accessed.

Is there any way to slow down bandwidth on the LEDs, perhaps? We’re not running elaborate animations or anything so it doesn’t need to be high frame rate.

Unfortunately I’m not able to switch to using the mp3 board as we’ve already had PCBs made to mount the SD reader board. These were working fine for a while and some are seemingly ok with being on for a long time which is making me think it might be something wrong with my soldering/wiring. Though everything works fine before it crashes.

int readChunk() {
    int retVal = 0;
    if (wavFile.isOpen()) {
        uint16_t* wav = speaker.getBuffer();
        uint8_t   buf[BUFFERSIZE * 2 * wh.num_channels];

It looks like BUFFERSIZE is 1024 and num_channels is dependent on the file, but usually 1 or 2, so that’s 2048 or 4096 bytes. For a stereo wav file, it’s 4K which is dangerously close to the 6K stack size. I’d guess you may be overwriting the end of the stack, which would cause that behavior. I’d move that buf to a global variable instead of allocating it on the stack.

1 Like

Interesting. Could this happen without even playing any files? I’m getting crashes before any files are accessed.

@ScruffR @rickkas7 sorry to ping again but I'm still having issues with this. Any insights into whether or not the SD card needs to be accessed / playing files in order for the issue to arise?

This would require playback to occur first though, right? I'm not getting any crashes on playback, only when the device is idle. Or, at least, crashes may occur on playback, but I haven't seen that yet.

@pselodux, this code in loop() will publish every 16ms (delayed by FastLED.delay()) until an incoming subscription sets device_name. In addition, FastLED.delay() being set to 16ms will delay the end of loop(), where the subscription handler gets called. The multi-publish and the delay may be your problem.

   if(device_name == "none"){
        Particle.publish("particle/device/name");
    }

For the publish(), you should have a small FSM that calls publish() then waits a few seconds before trying again.

That FastLED.delay() is more for an Arduino where there is no RTOS then for Particle. You may want to rethink your loop() code to be fully non-blocking and simply call FastLED().show() every 16ms using a millis() delay.

1 Like

Thanks for the suggestions, I’ll give them a try.

I just tested the Photon on its own, with no SD board attached and I’m still getting crashes, so I think it’s probably nothing to do with the SD card access. I’ll see if I can tighten up the LED/publish code as you suggested. I think given that we’re not running animations on the LEDs (yet!) we don’t need to call FastLED().show() so often anyway.

1 Like

This may be way off base, but you have a lot of hardware on your 5 volt supply. Since the lockups are random, is it possible that you don’t have enough decoupling capacitors on your PCB? You should have at least one 0.1 uF cap right by the 5 volt power pin of every digital IC on your board (particularly the SD card), and at least one decoupling capacitor by each connector for external (to the PCB) devices such as the amp and the LED strip. You can’t overkill keeping noise off of the logic power.

I did think that could potentially be the cause, but it still happens when I isolate the Photon and power it via USB with nothing connected.

So far so good. I removed the call to FastLED().show() and FastLED().delay() in the code and replaced it with:

    EVERY_N_MILLISECONDS(40)
        FastLED.show();
    }

It's been over two hours now without a freeze. I'll leave it overnight and see how it goes.

1 Like

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.