Creating BIG strings

Getting Started
1

HI. I’m wondering if someone can help me with a simple problem.

I want to track the time that a button is pressed. it might get pressed upwards of 200 times in a 5 minute period.
I have a demo working that transfers the time stamp of each button press using Spark.publish
But, if there is any network downtime, etc. I’d in theory miss the timestamps from button push events.

So, I’m thinking I’d like to save all button press time stamps locally, then publish a big comma delimited string of timestamps that I can parse on the browser side.

I’ve tried two methods with little success. For the below examples, just assume that a button press event is triggering everything inside the loop.

Method 1:

String all;{}

void setup();{

void loop();

all += “,”;
all += Time.timeStr();
Spark.publish(“Uptime”, all);

}

Method 2:

String all;
char publishTime[2000];

void setup();{}

void loop();{

all += “,”;
all += Time.timeStr();
all.toCharArray(publishTime,2000);
Spark.publish(“Uptime2”, publishTime);

}

Both of these methods works for 2.5 time stamps, returning something like this on the browser side:
,Sat Jun 21 15:23:47 2014 ,Sat Jun 21 15:23:52 2014 ,Sat Jun 21

Finally, I used the above two methods because nothing I do with Time.now(); seems to work.
Ideally, I’d like to use Time.timeStr(); as my timestamp because it seems simpler and I can manipulate it on the browser side easily.

Any help will be appreciated! @bko

2

@the_root_of_matt, the maximum string size you can publish is 63 characters. You can either break up your string into multiple “publishes” or use a different method like TCPClient to name one. :smile:

1 Like
3

Hi,

You can try my recently published library flashee - it provides various ways to access the flash, including a circular buffer aka. queue.

When the button is pressed you can push data into the queue. Later when you are ready to publish every 5 minutes or so, you can pull data out of the queue (say up to 50 timestamps) and publish them as a block.

If it’s only 200 presses every 5 minutes, then maybe that’s a small enough amount of data to store in a buffer in RAM - and store time as a timestamp, not as the string representation.

e.g.

    uint32_t timestamps[200];
    uint8_t index = 0;
    uint32_t publishTime;
    void setup() {
        publishTime = millis()+(60*5*1000);   // 5 mins
    }    

    void loop() {
       // detect button press
       timestamps[index++ % 200] = Time.now();
    
       if (publishTime<millis()) { 
          publishTime += 5*60*1000; 
          if (index) {
             String all = Time.timeStr(timestamps[0]);      
             for (int i=1; i<index; i++) {
                  all += ",";   all.concat(time[i]-time[i-1]);
             }
             Spark.publish("Uptime2", all.c_str());
             index = 0;          
         }    
      }
    }

To save space on the string, we only output the first time, and then the number of milliseconds difference between later times.

NB: you don’t need to use toCharArray, since this then means you have two copies of the data. You have only about 4-6k of memory to play with so reducing memory use is a good idea.

[And as a further aside, Ideally, publish() should take a Printable so that you can have objects stream themselves as a string without requiring the whole string representation to be held in memory, reducing memory requirements in these cases where we are sending large amounts of data. But that’s one for the spark team.]

EDIT: Note that you will still have to output far less than 200 timestamps in call to publish, due to the size limitations that @peekay123 mentioned above.

1 Like
4

Nice code :slight_smile:

Should this be if ((index % 200) == 0) { or if (index >= 200) { ?

Also I wonder about the millis() rollover case... I know subtraction with unsigned int is self correcting.

void setup() {
  publishTime = millis();
}

//...
if ((millis() - publishTime) > (60*5*100)) { 
          publishTime = millis();
//...
5

The if (index) tests if there were any button presses to publish. The % 200 stuff was just to write to the array with rollover, just to make the example simple (avoids the error checking on the bounds, but this might not be the desired behaviour since the results are no longer in chronological order if there are more than 200 button presses.)

You’re right about the millis() rollover - I was going to write it out like that but got lazy…
I figured it was just meant to be a sketch of the general idea, not bullet-proof! :slight_smile: But a valid point - timing against millis() should always be done the way you coded it there.

6

oh I gotcha now, it's just publishing whatever it has, up to 200 records... then resetting the index to 0 (effectively clearing the buffer without zeroing out the memory).

and I really like this technique timestamps[index++ % 200] :wink:

Not a bad idea, so it would work like essentially streaming the data via TCP every time Spark.publish() is called, instead of just scheduling it for sending after user code passes control to the BG tasks? This is kind of how Spark.publish() is suppose to work now, although I'm not sure about the BG tasks part... or if it's on demand (i.e. blocking user code). Either way, the limitation is then the RATE LIMIT of the server. Right now it's 60 per minute, up to 4 times per second. If that could be more dynamic like allow a higher rate with less data, then you could just Spark.publish("pressed", Time.now()); //every button press

Network downtime is a much bigger issue though, because the Spark Core can't run user code currently when the Core is trying to reconnect to the Cloud. It blocks user code. So there's (no?) way to currently save those button presses if you fell off the network. There are plans to make that process non-blocking though.... maybe? I forget if the CC3000 has an inherent issue that prevents this from being possible or not.

7

Yes, when you call Spark.publish(name, Printable), the printable writes it’s data to a print stream. In this case, it could be passed a small array of timestamps as part of that Printable class instance. When Printable.printTo() is called, these timestamps are then iterated over and printed to the Print instance given.

It’s a bit more convoluted than just writing a string directly, but the main gain being that there is no large buffer allocated - like you say, the object streams it’s string representation directly to the network stream. I imagine the spark team had streaming in mind when they added the Printable interface.

8

@mdma, that Printable thing is very interesting and clever. Any chance you can put together a simple example cause I think that should go under tips&tricks? A lot of people keep asking the same string questions, mostly pertaining to Spark.publish().

9

Thanks, but to make an example work, the Spark.publish() function would need upgrading to take a Printable as well as a const char*. When I wrote the code, I didn’t realize the maximum limit on publish data was under 100 bytes, in which case we might as well just pass in the full string.

But if that publish limit could be lifted, then using a Printable would be a way to pump out a large message (say a few Kb) without using a lot of memory. For example, the Printable could fetch data from flash memory and publish that.

10

@mdma, that sounds like an issue needs to be filed cause I’m not sure how much work is involved.

11

@mdma and @BDub and @peekay123
Thank you for the insight here. You’ve got me fundamentally re-considering my approach.
Now, You are all operating at a level way over my head, so as an initial foray into better understanding your feedback, I simply duplicated MDMA’s code with BDub’s mods as below:

Instead of a button, I just set it to run the loop every 5 seconds. It is not compiling however. Any stupid things I missed?

#include "flashee-eeprom/flashee-eeprom.h"

uint32_t timestamps[50];
uint8_t index = 0;
uint32_t publishTime;

unsigned long lastTime = 0UL;

void setup() {
        publishTime = millis();
    }    

void loop() 
{
unsigned long now = millis();    
if (now-lastTime>5000UL) 
    {
        lastTime = now;
        
        timestamps[index++ % 50] = Time.now();

            if ((millis() - publishTime) > (60*5*100)) 
            { 
            publishTime = millis();
          
          if (index) 
                {
             String all = Time.timeStr(timestamps[0]);      
             for (int i=1; i<index; i++) 
                    {
                  all += ",";   all.concat(time[i]-time[i-1]);
                    }
             Spark.publish("Uptime2", all.c_str());
             index = 0;          
                }    
            }
    }
}
12

@the_root_of_matt, the error output of the compiler would help helping.

13

That should read

all.concat(timestamps[i]-timestamps[i-1]);

(It was a typo in my original post - I saw it, but got sidetracked with the ongoing discussion.)

Also you don't need the flasheee library for that example, since the data is small enough to fit into RAM.

14

Ok, thank you. I modified my code as below. It resulted in the compile error at the bottom.

uint32_t timestamps[50];
uint8_t index = 0;
uint32_t publishTime;

unsigned long lastTime = 0UL;

void setup() {
        publishTime = millis();
    }    

void loop() 
{
unsigned long now = millis();    
if (now-lastTime>5000UL) 
    {
        lastTime = now;
        
        timestamps[index++ % 50] = Time.now();

            if ((millis() - publishTime) > (60*5*100)) 
            { 
            publishTime = millis();
          
          if (index) 
                {
             String all = Time.timeStr(timestamps[0]);      
             for (int i=1; i<index; i++) 
                    {
                  all += ",";   all.concat(timestamps[i]-timestamps[i-1]);
                    }
             Spark.publish("Uptime2", all.c_str());
             index = 0;          
                }    
            }
    }
}


In file included from ../inc/spark_wiring.h:30:0,
from ../inc/application.h:29,
from /recco_from_web.cpp:3:
../../core-common-lib/SPARK_Firmware_Driver/inc/config.h:12:2: warning: #warning "Defaulting to Release Build" [-Wcpp]
#warning "Defaulting to Release Build"
^
In file included from ../inc/spark_wiring.h:37:0,
from ../inc/application.h:29,
from /recco_from_web.cpp:3:
../inc/spark_wiring_ipaddress.h: In member function 'IPAddress::operator uint32_t()':
../inc/spark_wiring_ipaddress.h:53:52: warning: dereferencing type-punned pointer will break strict-aliasing rules [-Wstrict-aliasing]
operator uint32_t() { return *((uint32_t*)_address); };
^
../inc/spark_wiring_ipaddress.h: In member function 'bool IPAddress::operator==(const IPAddress&)':
../inc/spark_wiring_ipaddress.h:54:72: warning: dereferencing type-punned pointer will break strict-aliasing rules [-Wstrict-aliasing]
bool operator==(const IPAddress& addr) { return (*((uint32_t*)_address)) == (*((uint32_t*)addr._address)); };
^
../inc/spark_wiring_ipaddress.h:54:105: warning: dereferencing type-punned pointer will break strict-aliasing rules [-Wstrict-aliasing]
bool operator==(const IPAddress& addr) { return (*((uint32_t*)_address)) == (*((uint32_t*)addr._address)); };
^
/recco_from_web.cpp: At global scope:
/recco_from_web.cpp:3:9: error: 'uint8_t index' redeclared as different kind of symbol
#include "application.h"
^
In file included from /opt/gcc_arm/arm-none-eabi/include/stdlib.h:11:0,
from ../../core-common-lib/CC3000_Host_Driver/cc3000_common.h:43,
from ../../core-common-lib/SPARK_Firmware_Driver/inc/hw_config.h:35,
from ../inc/main.h:37,
from ../inc/spark_utilities.h:30,
from ../inc/spark_wiring.h:34,
from ../inc/application.h:29,
from /recco_from_web.cpp:3:
/opt/gcc_arm/arm-none-eabi/include/string.h:55:8: error: previous declaration of 'char* index(const char*, int)'
char *_EXFUN(index,(const char *, int));
^
/recco_from_web.cpp: In function 'void loop()':
/recco_from_web.cpp:19:25: error: ISO C++ forbids incrementing a pointer of type 'char* (*)(const char*, int)' [-fpermissive]
unsigned long now = millis(); 
^
/recco_from_web.cpp:19:25: error: lvalue required as increment operand
/recco_from_web.cpp:25:20: warning: the address of 'char* index(const char*, int)' will always evaluate as 'true' [-Waddress]

^
/recco_from_web.cpp:28:30: error: ISO C++ forbids comparison between pointer and integer [-fpermissive]
publishTime = millis();
^
/recco_from_web.cpp:33:20: error: assignment of function 'char* index(const char*, int)'
for (int i=1; i<index; i++) 
^
/recco_from_web.cpp:33:20: error: cannot convert 'int' to 'char*(const char*, int)' in assignment
make: *** [/recco_from_web.o] Error 1
15

It is important to me to store the timestamps locally on the Spark Core because I can’t guarantee I’ll be available to monitor when someone is using the device (pushing the button). I need to be able to choose to stream the data from a usage event that may have happened several hours ago.

@peekay123 Given the string size limitation, would it be advisable to put all of my timestamps into an array?

I think that if an array could handle 400 timestamps, I could be sure that I’ve given enough capacity to capture all button pushes in every usage scenario.

I am wondering if I could use a for() loop to rapidly change the array position being fed into the Spark.publish command.

Then I can re-build the array in Javascript.
If I had a lot of concerns about data integrity, I could perform this dump & rebuild twice, do a data comparison to see if I returned the same number of timestamps and if they were positionally the same between each dump&rebuild sequence.

16

@the_root_of_matt, I have to ask… can you not use TCPClient to send a the larger packet to your server instead of depending on multiple publishes? It could be triggered by a Spark.function() to send its data. Or a tinywebser that your server calls to pull the data. Either way, you would pull the data on request and send a reset data command/function when the data is successfully assembled. Perhaps @bko has some thoughts on this?

As for the array, if you button presses are essentially a button ID and a timestamp then it would look like this:

uint8_t  ButtonID       //1 byte
unsigned long timeStamp  //4 bytes

So for 400 entries at 5 bytes each that makes for a 2000 byte array which may or may not be doable depending on how much RAM the rest of you code takes.

1 Like
17

Since Spark.function() returns an int, you could call that over and over again and "pop" the oldest timestamp off the top of the array. You could loop this until your array is empty and returns something like 0 or -1 to show it's empty. That may or may not be preferable to using a TCPServer or TCPClient method, but it's another option to consider.

18

@wgbartley, I’m just wondering if doing 32+ Spark.publish to get the data accross is the best way, especially since you can only do one per second. Even with a burst of 4 in one second, that’s still 6+ seconds to get the (max) data!

19

I get that the amount of time stamps i want to retain will depend on my primary ram usage, and I may be pushing the limit. No way around that other than more elegant use of ram, or some external sd etc.

In terms of tcpclient, I just have no experience with this, so I’ll have to try to find an example.

If you know of an example I should check out, I’d definitely appreciate A point in the right direction.

20

If you’re pushing RAM limits, then you can store the timestamps in flash using the flashee library.

The key thing you need to figure out is how to stream the data, rather than trying to create a large buffer and sending it all. If you use a TCPClient then you can stream the data, using very little RAM.

Here’s a outline (this is not complete):

using namespace flashee-eeprom;

FlashDevice* device;
TCPClient client;   // connect the client to a server, see the spark docs for details
flash_addr_t addr;  // address to write to in flash for the next buttonID/timestamp
uint32 publishTime; // next time to publish
void setup() {
   device = Devices::createAddressErase();
   publishTime = millis();
}

void loop() {
     uint8_t button = detectIfButtonPressed();
     if (button) {   // <> 0 if a button was pressed
          // store the buttonID and time in flash
          uint32_t time = Time.now();
          flash->write(&button, addr++, 1);
          flash->write(&time, addr, sizeof(time));
          addr += sizeof(time);
     }

     if (!Wifi.on()) return; // don't bother publishing if no net connection
     // you may need to reconnect the TCPClient after network failure?

     // ok, we have net access, check if publish needed
     if ((millis() - publishTime) > (60*5*1000)) { 
            publishTime = millis();

            for (flash_addr_t a=0; a<addr; a+=5; ) {
                  uint8_t button;
                  uint32_t time;
                  flash->read(&bufton, addr, 1);
                  flash->read(&time, addr+1, 4);
                  // now stream the data over a socket - format this as you need to 
                  // this will write each button press as buttonID, time on a separate line
                  client.print(button);
                  client.print(',');
                  client.println(Time.timeStr(time));
            }
            addr = 0;
     }
}

The essence here is that the loop writes data to flash until it can be flushed over the network.

2 Likes