Xenon based BLE peripheral sensor Panic issue

I am using the BLE Group library that @mariano has developed to utilise the defunct but otherwise neat Xenon to create a BLE connected sensor group - in this case the sensor is an I2C interfaced AMG8833 a thermal array sensor that I am using to sense body temperature. I have migrated the Mesh network pub-sub to use the BLE Group. This has worked really well and the 3 sensors can be switched and start working with results going to a gateway device and off to the cloud.

The ‘however’ is that 2 of the 3 peripheral devices will stop working after a period and SOS 1 red flash. The third device can carry on until the battery is used up - several days. They are all running the same application software and device OS (1.4.4). Looking at the hard fault reasons it could be; invalid pointer, memory corruption or I2C wire call before Wire.begin() called.

The peripheral devices do a lot of sleeping and waking and can be woken interrupt from the AMG8833 which is put in a background checking whilst the Xenon is sleeping and can wake the Xenon to resume more intensive and nuanced reading and processing of the thermal camera array. There is also a RTC (DS3231) in the sensor.

I am looking for any advice about how to approach fault finding in this case, I am assuming it is I2C issue? One thing I don’t understand is that the only way to stop a device hard faulting is to remove the power and re-applying the power will see it continue to flash 1 red. A recovery requires the application to be reflashed.

@armor, posting some code and a block diagram of the hardware might help. Are you doing any I2C calls in the waking ISR? What sleep mode do you use? You may have an unchecked pointer causing writes where they shouldn’t be.

I just tried to swap the xenons in the working one and one of the others, with both flashed red straight away and also both when returned to the original state - so now all 3 have the issue!

The Sleep part of the code is like this:

                case S_DEV_SLEEP:                                       //device in sleep if wake on interrupt is rtc rather than pin then do the normal keep awake cycle
                    if (param.sleepPeriod > 0)
                    {
                        System.sleep(interruptPin, FALLING, param.sleepPeriod);
                        SleepResult result = System.sleepResult();
                        switch (result.reason())
                        {
                            case WAKEUP_REASON_PIN:
                                isNominal = true;
                                sensorMode = S_INT_WAKE;
                                break;
                            case WAKEUP_REASON_RTC:
                                isNominal = true;
                                sensorMode = S_RTC_WAKE;
                                break;
                            default:
                                isNominal = true;
                                sensorMode = S_INT_WAKE;
                        }
                    }
                    else
                    {
                        sensorMode = S_INT_WAKE;
                    }
                    break;
                case S_INT_WAKE:                                        //interrupt received whilst in sleep or whilst doing normal keep awake cycle, put sensor back into normal mode
                    makeSensorNormalMode();
                    sensorMode = S_EVERY_2S;
                    break;
                case S_RTC_WAKE:                                        //RTC wake from sleep, put sensor back into normal mode
                    makeSensorNormalMode();
                    checkBattery();                                     //get battery voltage/power state to send in heartbeat
                    tmpC = rtcsync.tempC();                             //read temperature from the DS3231
                    rtcsync.loop();                                     //resync the soft RTC time with the RTC
                    publishHeartbeat();                             //publish heartbeat
                    delay(200);
                    publishWait4Cmd();                              //publish wait4cmd
                    waitStart = millis();
                    sensorMode = S_WAIT4CMD;
                    break;

@armor, how are you pumping the FSM - from loop(), from a Software Timer? Is the AMG library using SPItransactions?

The FSM is running in loop() - there are actually 2 FSMs - there is a generic sensor FSM and a specific FSM for this sensor. The specific FSM for this sensor is run to handle the case of sleeping the Xenon whilst the sensor is waking every 10s and checking for thresholds then either interrupting which wakes the Xenon and continues with intensive reading - once every 1-2s or waiting until the end of the sleep period using the Xenon timer. No SPI both AMG8833 and DS3231 use I2C.