Spark core won't allow a reflash after I load this code

jaysettle · August 31, 2014, 6:28pm

Spark Core wont allow another flash after I load this code. The code pushes data streams to Plotly.com at your specified intervals. I have seen this code work. Sometimes it runs and I see a graph on Plotly:

Other times I don’t see the graph and not sure whats going on with the spark. I end up hard resetting the spark by holding the mode button down and then pressing the rst button and then still holding the mode button down for 10 more seconds. I then reconnect the spark with my iphone confirm the name by hitting ok. It goes through the flashes correctly and signs on my wifi network(breathing cyan). Also after this reset I’d say it’ll take a flash upload probably around 60% of the time.

There are three files here. An ino, a cpp, and a h file.

First the .ino file

// This #include statement was automatically added by the Spark IDE.
#include "plotly_spark.h"

#define DHTPIN A5
#define TEMPPIN A7
#define SOUNDPIN A4

#define NUM_TRACES 1

int t = 0.0;
float tempC = 0;
float tempF = 0;
int previousMillis = 0;
int interval = 10000;
int count = 0;

char szInfo[64];

char* streaming_tokens[NUM_TRACES] = {"w4ww1sdclw"};
plotly graph = plotly("jaysettle", "d4fer7wb60", streaming_tokens, "oldhag", NUM_TRACES);

//replace the tokens streamtoken, username, apikey and filename with corresponding values from Plotly settings.

// void message(std::string m1,
//      std::string m2="", std::string m3="", std::string m4="");


// Publush event
void Publish(char* szEventInfo) {
  Spark.publish("plotlyinfo", szEventInfo);
}

void setup() {
    graph.init();
    graph.openStream();

    Serial.begin(9600);
    digitalWrite(7, HIGH);
    pinMode(7, OUTPUT);
    pinMode(11, OUTPUT);
    pinMode(12, OUTPUT);
    pinMode(13, OUTPUT);
    pinMode(14, OUTPUT);
    pinMode(TEMPPIN, INPUT);
}

//float ConvertRawDataToTempF(int raw){
//    float tempC = 0.0;
//    float Result = 0.0;

//    tempC = (((raw * 3.3)/4095) - 0.5) * 100;
//    Result = (tempC * 9.0/5.0) + 32.0;

//    return Result;
//}

void loop() {
    digitalWrite(11, HIGH);
    digitalWrite(13, LOW);
  unsigned long currentMillis = millis();

  if (currentMillis - previousMillis > interval) {

    previousMillis = currentMillis;
    //    t = ConvertRawDataToTempF(analogRead(TEMPPIN));
    t = analogRead(TEMPPIN);

    tempC = (((t * 3.3) / 4095) - 0.5) * 100;
    tempF = (tempC * 9.0 / 5.0) + 12.0;

    sprintf(szInfo, "Temperature=%.2f °F", tempF);

    // Serial.println("%s °F", tempF);

    graph.plot(millis(), tempF, streaming_tokens[0]);

    Publish(szInfo);
    
    //digitalWrite(11, LOW);
  }


  Serial.print(tempF);
  Serial.print("    ");
  Serial.println(count);
  count++;
  delay(500);

}

The .cpp file:

#include "plotly_spark.h"


plotly::plotly(char *username, char *api_key, char* stream_tokens[], char *filename, int nTraces)
  {
    log_level = 2;  // 0 = Debugging, 1 = Informational, 2 = Status, 3 = Errors, 4 = Quiet (// Serial Off)
    dry_run = false;
    username_ = username;
    api_key_ = api_key;
    stream_tokens_ = stream_tokens;
    filename_ = filename;
    nTraces_ = nTraces;
    maxpoints = 30;
    fibonacci_ = 1;
    world_readable = true;
    convertTimestamp = true;
    timezone = "America/Montreal";
    fileopt = "overwrite";
}

bool plotly::init(){
    //
    //  Validate a stream with a REST post to plotly
    //
    if(dry_run && log_level < 3){
        Serial.println(F("... This is a dry run, we are not connecting to plotly's servers..."));
    }
    else if(log_level < 3) {
        Serial.println(F("... Attempting to connect to plotly's REST servers"));
    }
    while ( !client.connect("plot.ly", 80) ) {
        if(log_level < 4){
            Serial.println(F("... Couldn\'t connect to plotly's REST servers... trying again!"));
        }
        fibonacci_ += fibonacci_;
        delay(min(fibonacci_, 60000));
    }
    fibonacci_ = 1;
    if(log_level < 3){} Serial.println(F("... Connected to plotly's REST servers"));
    if(log_level < 3){} Serial.println(F("... Sending HTTP Post to plotly"));
    print_(F("POST /clientresp HTTP/1.1\r\n"));
    print_(F("Host: 107.21.214.199\r\n"));
    print_(F("User-Agent: Arduino/0.5.1\r\n"));

    print_(F("Content-Length: "));
    int contentLength = 126 + len_(username_) + len_(fileopt) + nTraces_*(87+len_(maxpoints)) + (nTraces_-1)*2 + len_(filename_);
    if(world_readable){
        contentLength += 4;
    } else{
        contentLength += 5;
    }
    print_(contentLength);
    // contentLength =
    //   44  // first part of querystring below
    // + len_(username)  // upper bound on username length
    // + 5   // &key=
    // + 10  // api_key length
    // + 7  // &args=[...
    // + nTraces*(87+len(maxpoints)) // len({\"y\": [], \"x\": [], \"type\": \"scatter\", \"stream\": {\"token\": \") + 10 + len(\", "maxpoints": )+len(maxpoints)+len(}})
    // + (nTraces-1)*2 // ", " in between trace objects
    // + 22  // ]&kwargs={\"fileopt\": \"
    // + len_(fileopt)
    // + 16  // \", \"filename\": \"
    // + len_(filename)
    // + 21 // ", "world_readable":
    // + 4 if world_readable, 5 otherwise
    // + 1   // closing }
    //------
    // 126 + len_(username) + len_(fileopt) + nTraces*(86+len(maxpoints)) + (nTraces-1)*2 + len_(filename)
    //
    // Terminate headers with new lines
    print_(F("\r\n\r\n"));

    // Start printing querystring body
    print_(F("version=2.3&origin=plot&platform=arduino&un="));
    print_(username_);
    print_(F("&key="));
    print_(api_key_);
    print_(F("&args=["));
    // print a trace for each token supplied
    for(int i=0; i<nTraces_; i++){
        print_(F("{\"y\": [], \"x\": [], \"type\": \"scatter\", \"stream\": {\"token\": \""));
        print_(stream_tokens_[i]);
        print_(F("\", \"maxpoints\": "));
        print_(maxpoints);
        print_(F("}}"));
        if(nTraces_ > 1 && i != nTraces_-1){
            print_(F(", "));
        }
    }
    print_(F("]&kwargs={\"fileopt\": \""));
    print_(fileopt);
    print_(F("\", \"filename\": \""));
    print_(filename_);
    print_(F("\", \"world_readable\": "));
    if(world_readable){
        print_("true");
    } else{
        print_("false");
    }
    print_(F("}"));
    // final newline to terminate the POST
    print_(F("\r\n"));

    //
    // Wait for a response
    // Parse the response for the "All Streams Go!" and proceed to streaming
    // if we find it
    //
    char allStreamsGo[] = "All Streams Go!";
    char error[] = "\"error\": \"";
    int asgCnt = 0; // asg stands for All Streams Go
    char url[] = "\"url\": \"http://107.21.214.199/~";
    char fid[4];
    int fidCnt = 0;
    int urlCnt = 0;
    int usernameCnt = 0;
    int urlLower = 0;
    int urlUpper = 0;
    bool proceed = false;
    bool fidMatched = false;

    if(log_level < 2){
        Serial.println(F("... Sent message, waiting for plotly's response..."));
    }

    if(!dry_run){
        while(client.connected()){
            if(client.available()){
                char c = client.read();
                if(log_level < 2) Serial.print(c);

                //
                // Attempt to read the "All streams go" msg if it exists
                // by comparing characters as they roll in
                //

                if(asgCnt == len_(allStreamsGo) && !proceed){
                    proceed = true;
                }
                else if(allStreamsGo[asgCnt]==c){
                    asgCnt += 1;
                } else if(asgCnt > 0){
                    // reset counter
                    asgCnt = 0;
                }

                //
                // Extract the last bit of the URL from the response
                // The url is in the form http://107.21.214.199/~USERNAME/FID
                // We'll character-count up through char url[] and through username_, then start
                // filling in characters into fid
                //

                if(log_level < 3){
                    if(url[urlCnt]==c && urlCnt < len_(url)){
                        urlCnt += 1;
                    } else if(urlCnt > 0 && urlCnt < len_(url)){
                        // Reset counter
                        urlCnt = 0;
                    }
                    if(urlCnt == len_(url) && fidCnt < 4 && !fidMatched){
                        // We've counted through the url, start counting through the username
                        if(usernameCnt < len_(username_)+2){
                            usernameCnt += 1;
                        } else {
                            // the url ends with "
                            if(c != '"'){
                                fid[fidCnt] = c;
                                fidCnt += 1;
                            } else if(fidCnt>0){
                                fidMatched = true;
                            }

                        }
                    }
                }
            }
        }
        client.stop();
    }

    if(!dry_run && !proceed && log_level < 4){
        Serial.println(F("... Error initializing stream, aborting. Try again or get in touch with Chris at chris@plot.ly"));
    }

    if(!dry_run && proceed && log_level < 3){
        Serial.println(F("... A-ok from plotly, All Streams Go!"));
        if(fidMatched){
            Serial.print(F("... View your streaming plot here: https://plot.ly/~"));
            Serial.print(username_);
            Serial.print(F("/"));
            for(int i=0; i<fidCnt; i++){
                Serial.print(fid[i]);
            }
            Serial.println(F(""));
        }
    }
    return proceed;
}
void plotly::openStream() {
    //
    // Start request to stream servers
    //
    if(log_level < 3){} Serial.println(F("... Connecting to plotly's streaming servers..."));
    char server[] = "arduino.plot.ly";
    int port = 80;
    while ( !client.connect(server, port) ) {
        if(log_level < 4) Serial.println(F("... Couldn\'t connect to servers... trying again!"));
        fibonacci_ += fibonacci_;
        delay(min(fibonacci_, 60000));
    }
    fibonacci_ = 1;
    if(log_level < 3){} Serial.println(F("... Connected to plotly's streaming servers\n... Initializing stream"));

    print_(F("POST / HTTP/1.1\r\n"));
    print_(F("Host: arduino.plot.ly\r\n"));
    print_(F("User-Agent: Python\r\n"));
    print_(F("Transfer-Encoding: chunked\r\n"));
    print_(F("Connection: close\r\n"));
    if(convertTimestamp){
        print_(F("plotly-convertTimestamp: \""));
        print_(timezone);
        print_(F("\"\r\n"));
    }
    print_(F("\r\n"));

    if(log_level < 3){} Serial.println(F("... Done initializing, ready to stream!"));
}

void plotly::closeStream(){
    print_(F("0\r\n\r\n"));
    client.stop();
}
void plotly::reconnectStream(){
    while(!client.connected()){
        if(log_level<4) Serial.println(F("... Disconnected from streaming servers"));
        closeStream();
        openStream();
    }
}
void plotly::jsonStart(int i){
    // Print the length of the message in hex:
    // 15 char for the json that wraps the data: {"x": , "y": }\n
    // + 23 char for the token: , "token": "abcdefghij"
    // = 38
    if(log_level<2) Serial.print(i+44, HEX);
    if(!dry_run) client.print(i+44, HEX);
    print_("\r\n{\"x\": ");
}
void plotly::jsonMiddle(){
    print_(", \"y\": ");
}
void plotly::jsonEnd(char *token){
    print_(", \"streamtoken\": \"");
    print_(token);
    print_("\"}\n\r\n");
}

int plotly::len_(int i){
    // int range: -32,768 to 32,767
    if(i > 9999) return 5;
    else if(i > 999) return 4;
    else if(i > 99) return 3;
    else if(i > 9) return 2;
    else if(i > -1) return 1;
    else if(i > -10) return 2;
    else if(i > -100) return 3;
    else if(i > -1000) return 4;
    else if(i > -10000) return 5;
    else return 6;
}
int plotly::len_(unsigned long i){
    // max length of unsigned long: 4294967295
    if(i > 999999999) return 10;
    else if(i > 99999999) return 9;
    else if(i > 9999999) return 8;
    else if(i > 999999) return 7;
    else if(i > 99999) return 6;
    else if(i > 9999) return 5;
    else if(i > 999) return 4;
    else if(i > 99) return 3;
    else if(i > 9) return 2;
    else return 1;
}
int plotly::len_(char *i){
    return strlen(i);
}
void plotly::plot(unsigned long x, int y, char *token){
    reconnectStream();
    jsonStart(len_(x)+len_(y));
    print_(x);
    jsonMiddle();
    print_(y);
    jsonEnd(token);
}
void plotly::plot(unsigned long x, float y, char *token){
    reconnectStream();

    char s_[15];
    dtostrf(y,2,3,s_);

    jsonStart(len_(x)+len_(s_)-1);
    print_(x);
    jsonMiddle();
    print_(y);
    jsonEnd(token);
}
void plotly::print_(int d){
    if(log_level < 2) Serial.print(d);
    if(!dry_run) client.print(d);
}
void plotly::print_(unsigned long d){
    if(log_level < 2) Serial.print(d);
    if(!dry_run) client.print(d);
}
void plotly::print_(float d){
    if(log_level < 2) Serial.print(d);
    if(!dry_run) client.print(d);
}
void plotly::print_(char *d){
    if(log_level < 2) Serial.print(d);
    if(!dry_run) client.print(d);
}
/*
void plotly::print_(const __FlashStringHelper* d){
    if(log_level < 2) Serial.print(d);
    if(!dry_run) client.print(d);
}
*/

//convert double to ascii
char *plotly::dtostrf (double val, signed char width, unsigned char prec, char *sout) {
  char fmt[20];
  sprintf(fmt, "%%%d.%df", width, prec);
  sprintf(sout, fmt, val);
  return sout;
}

And the .h file:

#ifndef plotly_streaming_spark_h
#define plotly_streaming_spark_h

#include "application.h"

class plotly
{
    public:
        plotly(char *username, char *api_key, char* stream_tokens[], char *filename, int nTraces);
        TCPClient client;
        bool init();
        void openStream();
        void closeStream();
        void reconnectStream();
        void jsonStart(int i);
        void jsonMiddle();
        void jsonEnd(char *token);
        char* dtostrf (double val, signed char width, unsigned char prec, char *sout);

        void plot(unsigned long x, int y, char *token);
        void plot(unsigned long x, float y, char *token);

        int log_level;
        bool dry_run;
        int maxpoints;
        bool world_readable;
        bool convertTimestamp;
        char *timezone;
        char *fileopt;

    private:
        void print_(int d);
        void print_(unsigned long d);
        void print_(float d);
        void print_(char *d);
        //void print_(const __FlashStringHelper* d);

        int len_(int i);
        int len_(unsigned long i);
        int len_(char *i);

        unsigned long fibonacci_;
        char *username_;
        char *api_key_;
        char** stream_tokens_;
        char *filename_;
        int nTraces_;

};
#endif

Not sure how long these credentials will work in the ino file below as I might generate new keys soon.

char* streaming_tokens[NUM_TRACES] = {"w4ww1sdclw"};
plotly graph = plotly("jaysettle", "d4fer7wb60", streaming_tokens, "oldhag", NUM_TRACES);

Also from messing with this code I’ve found that the Spark Core get real flaky when my uploading intervals are down to 2 or 3 seconds. I’m I seeing this right?

I can reflash continuously with this simple blinking led code:

void setup() {
    pinMode(7,OUTPUT);
    pinMode(11,OUTPUT);
    pinMode(12,OUTPUT);   
    pinMode(13,OUTPUT);
    pinMode(14,OUTPUT);
}

void loop() {
digitalWrite(7, HIGH);
delay(50);
digitalWrite(7, LOW);
delay(50);


digitalWrite(13, LOW);

digitalWrite(11, HIGH);
delay(50);
digitalWrite(11, LOW);
delay(50);

digitalWrite(12, HIGH);
delay(50);
digitalWrite(12, LOW);
delay(50);


digitalWrite(14, HIGH);
delay(50);
digitalWrite(14, LOW);
delay(50);

}

Thanks for any help!

bko · August 31, 2014, 8:57pm

Hi @jaysettle

Spark.publish() is rate-limited to an average of 1 per second with a burst of upto 4 allowed as long as the average is maintained. I think you are running into that limit since you main loop only delays for around 1/2 a second.

There are some problems where you code makes web flashing impossible without a factory reset, so I sometimes add this little piece of code at the top of my loop().

   // declare D0 as an input above.
    int pin = digitalRead(D0);
    if (HIGH==pin) {
        Serial1.print(" Waiting...");
        for(;;) {
        SPARK_WLAN_Loop();
        }
    }

This allows me to test my code with pin D0 tied low, but if I have trouble, I can tie D0 high and hit reset. Then the core will infinite loop running SPARK_WLAN_Loop() which processes all the Spark requests and allow you to flash new code. You can change it to use any pin.

jaysettle · August 31, 2014, 9:48pm

Publish is inside a loop that runs after 7 seconds is up. Thanks for the suggestion! It’s still really flaky though.

maxint · September 2, 2014, 2:58pm

Using the pin-check loop is a good suggestion. Does that D0 circuit include any resistors?

In my project I added a 20 second loop in the setup() function to allow for flashiing after reset. After that my Core gets disconnected from the cloud to improve stability…

bko · September 2, 2014, 3:41pm

I just move the wire on the bread board when I am debugging but you could add a pull-down resistor either externally or by changing the pinMode() call.

maxint · September 4, 2014, 12:41pm

@jaysettle: perhaps the OTA flash is restricted by the size of your code plus libraries used.

In the past two days I had problems with OTA flashing too. As I found out, in my case there appears to be a code-size related restriction. As soon as the code-size is about 97.7% or more, flashing doesn’t work. The IDE message is that flashing went okay, but the purple led-flashing doesn’t start. When encountering this limit, removing any piece of code helps. I reproduced this issue by adding and deleting debug-statements.

My Spark Core project uses a couple of handy libraries that increased the program size. My own code is less than a couple of KB. Now I’ve cleaned up my code a bit to allow for further development, but it’s a pity I can’t use the full 100%. Even calling a function like analogWrite will increase the size with a couple of percents, so I guess this call’s for more effective coding or maybe even stripping out unused code from the libraries…

peekay123 · September 4, 2014, 1:41pm

@maxint, the linker will take care of omitting any library code not used by your program. Which “handy” libraries are you using?

maxint · September 4, 2014, 1:51pm

Hi @peekay123. Thanks for your reply. I indeed experienced that library code is only included when used. When adding the call to analogWrite (for smooth led-lighting) the size went up more than that of just a single function call, so I guess that for analog out another library-part got added to the mix.

The handy libraries I use in my project are Webserver (which has plenty of code candidate for stripping out), Adafruit_mfGFX_IDE (without the extra fonts enabled yet) and the Spark port of Adafruit_PCD8544 (by pkourany , which is not published as a library yet). On top of that I added some custom code (excluding all comments probably perhaps 250 lines).

peekay123 · September 4, 2014, 1:56pm

@maxint, a classic killer space eater is sprintf which adds about 20KB of flash! Yes, adding some function calls will pull in all the dependent functions as well. To reduce the final size of your code you will need to optimize the code that is actually used and sometimes, that is not always easy if it’s in a library.

Nonetheless, the “maximum” size of code that can be flashed via OTA is of concern. Can you flash the same (97.7%) code via USB?

maxint · September 4, 2014, 2:09pm

@peekay123, thank you for your tips. (But thank you much more for the PCD8544 Nokia5110 port!)
BTW. Forgot to mention that I also use the SD-Card library.

I do use functions such as sprintf and vsprintf, but wouldn’t want to code without them.

I guess when required I need to make my own version of the Webserver library. That library contains some code I don’t use and some I even don’t want. (I.e. my webserver serves files from the uSD card, so that library-default icon is just wasting space!)

I haven’t installed the dfu-util yet. So far I managed to get where I want using the online IDE. There the experienced limit seems like a strange anomaly and I wouldn’t expect that from flashing using USB. It’s worth a try though…

peekay123 · September 4, 2014, 2:13pm

@maxint, personally I use the Spark CLI almost exclusively for all my coding. It gives me the most flexibility. I also have a local toolchain for testing unreleased versions of the core firmware.

maxint · September 4, 2014, 2:15pm

@peekay123: I just read here that on Windows the flashing utility is’n really straight-forward. On mbed I can just copy and paste the compiled firmware.bin. Do you know of something similar for Spark Core?

Edit: BTW. those libraries mentioned are in addition to those that I didn’t need to add, such as for the Servo class, which is likely to take space too. What bothered me most was that - as it seems - OTA flashing can break before reaching 100%. It took some hours before I realized that one was related to the other…

jaysettle · September 4, 2014, 2:27pm

I’m still having reflashing problems. I’m using the spark web app. I’m not uploading through usb at all. Could this be a problem with space requirements with just the little code I have pasted above. I hope not.

I’ve since claimed the spark core unreliable for over the air uploads. Please restore my faith.

maxint · September 4, 2014, 2:33pm

@jaysettle: what percentage do you see after using the Verify?
(To see click the (i)-icon behind “Ready” or “Code verified! Great work.”)

Does the IDE say that flashing was succesfull or not?

bko · September 4, 2014, 2:35pm

There is some interaction that I don’t understand between publishing and being able to flash OTA.

As a test @jaysettle can you comment out your actual Spark.publish() and see if you can reflash then?

I know this is not a perfect test but it will point in a general direction.

peekay123 · September 4, 2014, 3:19pm

@maxint, I am not having ANY problems using DFU-UTIL on Win 8.1 x64 and never have. I have dfu-util v0.7 and soon 0.8 once someone compiles it for windows and makes it available. I created a one-line batch file (flash.bat) to flash to the connected core:

dfu-util -d 1d50:607f -a 0 -s 0x08005000:leave -D %1

So I simply use "flash firmware.bin" when I compile locally to flash my code on the core.

bko · September 4, 2014, 3:23pm

There is a make target for flashing in the makefile so you can build and flash in one command line:

make program-dfu

There also a cloud flash target that uses curl:

make program-cloud

jaysettle · September 4, 2014, 3:25pm

I’m at work now but I can remember the IDE says unsuccessful. I get no indication from the core from watching the LEDs that it has received code over the air.

I will try to comment out “publish” when I return home.

Never heard of DFU-UTIL. Is this used in confuction with spark web app?

Thanks.

Hootie81 · September 4, 2014, 4:48pm

@peekay123,

@mdma did compile it a while ago "for testing" works a treat for me

Hootie81 · September 4, 2014, 5:59pm

Its a tiny program that lets you flash the core via USB. you can use it with the Spark CLI or by itself. It takes about 20seconds to transfer the code onto the core!

With the CLI its really simple, i use the cloud button next to the name in the Build web IDE to save firmware.bin into a folder, then run spark flash --usb firmware.bin from a cmd prompt

how to here : HOWTO: Update CC3000 via DFU-util (windows) - Troubleshooting - Particle

Topic		Replies	Views
Spark Core + Plotly Libraries	21	6747	November 12, 2015
Spark Will not re-flash when a Serial.begin() is present Troubleshooting	1	629	July 27, 2014
Can't Re-Flash Cores [SOLVED] Troubleshooting	19	4747	March 8, 2014
Can't flash my code to the Core Getting Started	5	1510	January 15, 2014
Spark Core not updating with new code Troubleshooting	13	2207	January 27, 2016

Spark core won't allow a reflash after I load this code

Related Topics