How to debug possible code issues on a deployed device

jonpasc · April 20, 2019, 3:00pm

I have two borons deployed in a semi accessible location (1 hour drive to visit) meant to do basic GPS tracking on a piece of equipment using adafruit ultimate GPS. These two devices are about 0.5 miles from each other and should be identical, but I’ve experienced some issues that I’m not even sure how to start diagnosing. Writing the code for these devices is the extent of my coding skills and knowledge. I’m looking for ideas about how to start diagnosing if my code is causing the devices to lock up or freeze up, or if it’s a power related problem. Both devices are in the exact same setup and firmware. **If you see an issue, great, but if you have recommendations of steps or diagnostic features I should research, also great! **

As a note, the power supply is probably not very clean and could be a problem, the equipment natively has a 480V transformer stepping down to 120V single phase. My device has a 120VAC to 12VDC converter off board and then a 12 to 5V converter circuit on the board. I have 2 other devices meant to control relays that have had no issue with almost the exact same power setup in weeks deployed in a slightly different piece of equipment.

Device 1- Worked fine for 3 days and then started continuously restarting (every 2 min) as indicated by the console event terminal showing restarts and diagnostics. It was counting up to 40 disconnects. We manually cut the power and let it restart and it worked perfectly. I would blame cell signal, but 6 hours later we moved the device in the same area with no issues and while this error was occurring, no issues occurred on the similar device .5 miles away (perfectly flat terrain, both devices record around 50,50 on strength and cell quality). The device has run since then >1 week with no issue.

Device 2 - Device 2 is restarting every 12-24 hours, and otherwise works as normal. With the same exact code, device 1 does not have this issue. To make it weirder, I believe it worked fine originally for a few days. We then reset the machine which flashed power.

I don’t know if I have an electrical issue with my base power supply, poor coding practices causing problems, or issues on the circuit board I built. My only idea to diagnose is to deploy a second device along side device 2 feeding on the same power and see if they both experience issues. What I’d like are some ideas on how to remotely diagnose code performance. Is there a way to track when a catastrophic issue occurs and send that once the device restarts.

I deployed all my prototype devices. I’m getting parts to build another one to sit at home where I can watch it, but I think the medium to poor cell service and power on site may not allow me to recreate what’s causing the issues.

Thank you for any help!

// This #include statement was automatically added by the Particle IDE.
#include <AssetTracker.h>

// This #include statement was automatically added by the Particle IDE.



// Used to keep track of the last time we published data
char dev_name[32] = "";
char msg[16] = " Restarted";
char startupMsg[50];
bool publishName = false;

uint32_t lastGPSUpdate = 0;
uint32_t alertSetTime = 0;

// How many minutes between publishes? 10+ recommended for long-time continuous publishing!
int delaySeconds = 2;


int latPivotAddr = 10;
int longPivotAddr = 30;
//float latPivot = EEPROM.read(latPivotAddr);
//float longPivot = EEPROM.read(longPivotAddr);
float latPivot = 34.941898;     //MCEast34.941898            //MCWest34.941953;
float longPivot = -102.337646;  //MCEast-102.337646          //MCEast-102.346287;
double latPivotReported; //Only need float for GPS, but cloud requries double as exposed variable.
double longPivotReported; //Only need float for GPS, but cloud requries double as exposed variable.
float latGPS;
float longGPS;

float latGPS30m =0;
float longGPS30m =0;

float d30m = 0;
int brng;
int alertBrng = 0;
bool alertState = false;
bool publishedAlert = false;
//bool thirtyMinElapsed = false;
bool timeToUpdateDistance = false;
bool publishFlag = false;
int publishFreq = 5; //min between publish
int alertTriggerDist = 10;
bool startUpComplete = false;
bool firstDistCaptured = false;


Timer timer30m(60000 * 30, distance30mFunc); // 60 sec * 30  = 30 min
Timer publishTimer(60000 * publishFreq, publishFunc);


float updateLongPivot(String command);
float updateLatPivot(String command);
int updateAlertBrng(String command);



// Creating an AssetTracker named 't' for us to reference
AssetTracker t = AssetTracker();


void setup() {
  
    Particle.function("PivotLong", updateLongPivot);
    Particle.function("PivotLat", updateLatPivot);
    Particle.function("MinDistAlrm", updateAlertTriggerDist);
    Particle.function("AlertState", updateAlertState);
    for (uint32_t ms = millis(); millis() - ms < 1000; Particle.process());  //adhere to data limits
    Particle.function("SetAlert", updateAlertBrng);
    Particle.variable("PivotLong", longPivotReported);
    Particle.variable("PivotLat", latPivotReported);
    Particle.variable("Bearing", brng);
    for (uint32_t ms = millis(); millis() - ms < 1000; Particle.process());  //adhere to data limits
    Particle.variable("AlertBearing", alertBrng);
    Particle.variable("AlertState", alertState);

    Particle.subscribe("particle/device/name", handler);
    Particle.publish("particle/device/name");
    for (uint32_t ms = millis(); millis() - ms < 3000 && !publishName && Particle.connected(); Particle.process());
    strcpy(startupMsg, dev_name);
    strcat(startupMsg, msg);
    
    Particle.publish("twilio_sms", startupMsg, PRIVATE);
    //Particle.publish("restart","restarted");
    //Particle.publish("twilio_sms_test", startupMsg, PRIVATE);
    
    // Sets up all the necessary AssetTracker bits
    t.begin();

    // Enable the GPS module. Defaults to off to save power.
    // Takes 1.5s or so because of delays.
    t.gpsOn();

    // Opens up a Serial port so you can listen over USB
//    Serial.begin(9600);
/*
    if(latPivot = 0xffffffff)
    {
        //EEPROM was empty -> initialize value
    latPivot = 34.986503;
    }
    
    if(longPivot = 0xffffffff)
    {
        //EEPROM was empty -> initialize value
    longPivot = -102.380795;
    }
*/
    
    latPivotReported = latPivot; //Only need float for GPS, but cloud requries double as exposed variable.
    longPivotReported = longPivot; //Only need float for GPS, but cloud requries double as exposed variable.

    uint32_t startTime = millis();

    publishTimer.start();
}


// loop() runs continuously
void loop() {
    // You'll need to run this every loop to capture the GPS output
    t.updateGPS();
    
    if(startUpComplete == false)
        {
            if(t.gpsFix() == true)
            {
                timer30m.start();
                startUpComplete = true;
                latGPS30m = t.readLatDeg();
                longGPS30m = t.readLonDeg();
            }
        }

    if (millis()-lastGPSUpdate > delaySeconds*1000) { //could probably run without delay, but kept in for no specific reason
        
        lastGPSUpdate = millis();
        latGPS = t.readLatDeg();
        longGPS = t.readLonDeg();

        brng = bearing(latPivot, longPivot, latGPS, longGPS);

        //determine if pivot has reached alert bearing set by user
        if((alertState == true) && (brng == alertBrng) && (publishedAlert == false)){ //possible issue, what if it's walking fast and the GPS jumps?
            //alertstate - only set alerts if allerttriggers are on, publishedAlert - only send alert message once, skip if this already happend  
            //for(;Particle.connected(); Particle.process());
            Particle.publish("twilio_sms", dev_name, PRIVATE);
            //Particle.publish("twilio_sms_test", dev_name, PRIVATE);
            publishedAlert = true;
            alertState = false;  //between alertState and publishedAlert, one of them is redundant
            for (uint32_t ms = millis(); millis() - ms < 500; Particle.process());
        }
        
        //deterine if pivot has moved (give pivot 30 min after startup before alarm will trigger)
        if (millis() - alertSetTime > 1000 * 1800){ //1800 secons = 30 min
            if((alertState == true) && (publishedAlert == false) && (d30m < alertTriggerDist) && (firstDistCaptured == true)){ //&& (thirtyMinElapsed = true)
                //for(;Particle.connected(); Particle.process());
                Particle.publish("twilio_sms", dev_name, PRIVATE);
                //Particle.publish("twilio_sms_test", dev_name, PRIVATE);
                publishedAlert = true;
                alertState = false;
                for (uint32_t ms = millis(); millis() - ms < 500; Particle.process());
            }
        }  
    }
    
    if (timeToUpdateDistance == true)
    {
        d30m = distance(latGPS30m,longGPS30m,latGPS,longGPS);
        latGPS30m = latGPS;
        longGPS30m = longGPS;
        //thirtyMinElapsed = true;
        timeToUpdateDistance = false;
        firstDistCaptured = true;
    }


    if(publishFlag == true)
    {
            CellularSignal sig = Cellular.RSSI();
            int strength = sig.getStrength();
            int quality = sig.getQuality(); 
        Particle.publish("Pivot", (String)latGPS + "," + (String)longGPS + ":" + (String)brng + ":" + (String)alertBrng + ":" + (String)d30m + ":" + (String)alertState + ":" + (String)strength + ":" + (String)quality);
        publishFlag = false;
    }
}
 
 
void publishFunc()
    {
        publishFlag = true;
    }
        
void handler(const char *topic, const char *data) {
  strncpy(dev_name, data, sizeof(dev_name)-1);
  publishName = true;
}

void distance30mFunc() { //triggers a check of the distance traveled ever 30 min, main loop resets to false after updating value
    timeToUpdateDistance = true;
}


float updateLongPivot(String command)
    {
        float temp;
        temp = command.toFloat();
        if ((temp < -180) || (temp > 180))
        {
            return -1;
        }
        else
        {
        longPivot = temp;
        //EEPROM.put(longPivotAddr, longPivot);
        longPivotReported = longPivot; //Only need float for GPS, but cloud requries double as exposed variable.
        return temp;            
        }
    }

float updateLatPivot(String command)
    {
        float temp;
        temp = command.toFloat();
        if ((temp < -90) || (temp > 90))
        {
            return -1;
        }
        else
        {
        latPivot = temp;
        //EEPROM.put(latPivotAddr, latPivot);
        latPivotReported = latPivot; //Only need float for GPS, but cloud requries double as exposed variable.
        return temp;            
        }
    }
    
int updateAlertBrng(String command)
    {
        int temp;
        temp = command.toInt();
        
        if((temp<0) || (temp > 360)){ //check that value is within the possible degrees range
            return -1;
        }
        
        if((temp>=0) && (temp<=359)){
            alertBrng = temp;
        }
        else{
            alertBrng = 0; //if the user enters 360, default to 0
        }
            alertState = true;
            publishedAlert = false;
            //thirtyMinElapsed = false;
            alertSetTime = millis();
            return alertBrng;    
    }

int updateAlertTriggerDist(String command)
    {
        int temp;
        temp = command.toInt();
        if((temp <= 0) || (temp > 1000))
            {
                return -1;
            }
        else
        {
            alertTriggerDist = temp;
            return temp;
        }
    }
    
int updateAlertState(String command)
    {
        if((command == "off") || (command == "false"))
        {
            alertState = false;
            return 1;
        }
        if((command == "on") || (command == "true"))
        {
            alertState = true;
            return 1;
        }
        return -1;    
        
    }

int bearing(float lat,float lon,float lat2,float lon2){

    float teta1 = radians(lat);
    float teta2 = radians(lat2);
    float delta1 = radians(lat2-lat);
    float delta2 = radians(lon2-lon);

    //==================Heading Formula Calculation================//

    float y = sin(delta2) * cos(teta2);
    float x = cos(teta1)*sin(teta2) - sin(teta1)*cos(teta2)*cos(delta2);
    float brng = atan2(y,x);
    brng = degrees(brng);// radians to degrees
    brng = ( ((int)brng + 360) % 360 );  //360 doesn't exist, it returns the value of 0
    return brng;
  }


int distance(float lat1,float long1,float lat2,float long2) {
    float d_ew = (long2-long1)*cos(radians(lat1)) * 69; //69 is the approximate distance between 
                                                        //longitude degrees at equator in statue miles, cosine approximates north
    float d_ns = (lat2-lat1) * 68.92; //68.92 is the approximate distance between latitude degrees at 34 degrees north.
    float d = sqrt(d_ew*d_ew + d_ns*d_ns) * 5280;
    
    return (int)d;
}


float degrees(float radians) {
    float drg;
    drg = radians * 180 / M_PI;
    return drg;
}

float radians(float degrees) {
    float rad;
    rad = degrees * M_PI / 180;
    return rad;
}

ScruffR · April 20, 2019, 3:18pm

Most points bellow probably won’t make your issue go away, but still should be addressed.

float updateLongPivot(String command);
float updateLatPivot(String command);
[...]
    Particle.function("PivotLong", updateLongPivot);
    Particle.function("PivotLat", updateLatPivot);

I’m surprised this code even compiles. Particle.function() callbacks are meant to return int not float
You should also rather use the AssetTrackerRK library as it is more stable.

AssetTracker t = AssetTracker();

would be better written this way (this already instatiates via the default constructor)

AssetTracker t;

Particle.publish("particle/device/name");

From that statement I deduce that you are not using the most recent device OS versions. In later versions (IIRC 0.8.0-rc.4+) it’s compulsory to provide the scope parameter (best use PRIVATE).

You have this …

    uint32_t startTime = millis();

… as last but one statement in setup(), but since this startTime is local, it doesn’t propagate beyond the scope of setup().

And as others might know, I can’t help pointing out that you shouldn’t use String but rather C strings (char[]).
e.g. here

        Particle.publish("Pivot", (String)latGPS + "," + (String)longGPS + ":" + (String)brng + ":" + (String)alertBrng + ":" + (String)d30m + ":" + (String)alertState + ":" + (String)strength + ":" + (String)quality);

snprintf() will help formatting your string (and again, you should scope this even PRIVATE.

In order to run your t.updateGPS() as often as possible, you may want to find some other way to ensure obeying the publish rate limit than these lines

  for (uint32_t ms = millis(); millis() - ms < 500; Particle.process());

e.g. setting a msRateLimit = 500; and msLastPublish = millis(); in your respective conditional blocks and then do this in loop()

void loop() {
  t.updateGPS();
  if (millis() - msLastPublish() < msRateLimit) return;
  msRateLimit = 0;
  ...
}

And as a plea for people who you’d like to read your code, could you try to use a more common indentation scheme
e.g. this doesn’t help readability

int updateAlertBrng(String command)
    {
        int temp;
        temp = command.toInt();
        
        if((temp<0) || (temp > 360)){ //check that value is within the possible degrees range
            return -1;
        }
        
        if((temp>=0) && (temp<=359)){
            alertBrng = temp;
        }
        else{
            alertBrng = 0; //if the user enters 360, default to 0
        }
            alertState = true;
            publishedAlert = false;
            //thirtyMinElapsed = false;
            alertSetTime = millis();
            return alertBrng;    
    }

jonpasc · April 22, 2019, 2:32pm

@ScruffR.
Particle.function float return
That was way more time and effort than I was looking for, so thank you very much. I’ll work through this list in the next week, but wanted to make a note on the float return of the particle function. Not only does a particle function with a float return compile, it works (ish). I actually spent a little time trying to figure out why my return values weren’t exactly what I typed in. e.g. for GPS, if I submitted 102.123456 as the command. The function should have stored that value to a variable and returned the exact value upon success of the function. In general I was getting 6 numerals correct and then randomness (input = 102.123456, return = 102.123593230203857 and input = 33.123456, return = 33.123434250098324). I spent time trying to examine how processors interpreted float values, but after a few hours decided I was getting at least 4 decimal places accurate which is within GPS sensor resolution already so I decided not to continue exploring.

Particle.publish scope paramater
I copied the Particle.publish(“particle/device/name”) out of one of your previous post which was very helpful(likely an older version recommendation). I’ve been using version 0.9.0 without compile or publish problems, but I’ll add in the scope parameter. This might be good to see on your side why it’s compiling without the required parameter.

Sorry about the code formatting, I’m self taught through just reviewing examples online and gloss over how code should look in a professional environment. I’ll try to use a common indention scheme.

Regarding the String/Char, I’ve read at least 10 of your post about using char over string and that was my guess as to my issue culprit. I keep trying to get Char to work, but continue to fail at that, so String is the short term method to get things moving. I’m trying to transition over, I just need more research and reading time to figure out correct usage to make it compile.

ScruffR · April 22, 2019, 2:53pm

@rickkas7, can you confirm and possibly update the docu for Particle.function() also allowing float as return value?
I've never (conciously) seen a change log entry about that (the same es with actual bool support for Particle.variable()).

The same would go for examples like this
https://docs.particle.io/reference/device-os/firmware/photon/#get-device-name
Which would throw errors with newer device OS versions

I think that's due to the fact that the Gen3 device OS was branched off of Gen1&2 version prior the introduction of that demand in 0.8.0-rc.4 (I think) so 0.9.0 Gen3 was in some features lagging behind 0.8.0-rc.x on Gen1&2 - not good style (I agree ), but 1.1.0 should consolidate these discrepancies.

This might help

  char msg[128];
  snprintf(msg, sizeof(msg)
          , "%.6f,%.6f:%d:%d:%.6f:%s:%d:%d"
          , latGPS
          , longGPS
          , brng
          , alertBrng
          , d30m
          , alertState ? "TRUE" : "FALSE",
          , strength
          , quality);
  Particle.publish("Pivot", msg, PRIVATE);

Topic		Replies	Views
Boron Unstable Behavior Cellular	4	460	September 9, 2022
Boron connectivity issues Cellular boron , boron-3g	1	313	January 12, 2024
Woes remote managing a boron Troubleshooting	6	1297	February 2, 2019
Random hard_fault on Boron Firmware	3	393	December 5, 2020
Debugging a locked up Boron Troubleshooting boron	5	1572	May 16, 2019

How to debug possible code issues on a deployed device

Related topics