PublishQueueAsyncRK and Large Datasets

TrikkStar · October 15, 2019, 3:08pm

I’m using the AsyncQueue library by @rickkas7, and am wondering what the best way to handle arbitrarily large amounts of data would be? I moved away from using a retained array as I don’t need the data to persist across a reset, and it allows me to get up to 7kb before my P1s stop responding (publish is made to the queue, but never shows up in the console), which is odd since I’ve done a 20kb array on a photon and it works just fine.

In theory I’d like to used something like SDFat or some other external storage library to store the data while the device is offline and then send it to the async queue to be pushed up to the cloud when the connection is reestablished. However I don’t really understand how the library works (looked at the source but I don’t understand what’s going on) and was wondering if anyone had some advice for incorporating a separate queue source?

armor · October 15, 2019, 5:04pm

@TrikkStar Before there was AsyncQueue library based upon retained memory I needed a solution so developed a solution using a standard .txt file on an SD card. I can’t share this as it was developed for a client. I can tell you what is required if you want to try yourself. The file is structured with 2 uint32_t (ie. 4 byte) indexes at the start of the file that point to the head of the queue and the tail of the queue. Uses \n as the end of record delimiter. When you secure Publish it takes the event name and event data and adds to the tail updating the index after it has written. Using a timer every second if there is an event message queued (head != tail) then read from head and if publish successful delete.

TrikkStar · October 21, 2019, 12:59pm

Thanks for the advice @armor. I’ll look into that, but a bigger issue I’m encountering is that when I include the SDFat lib in my dev branch I’m overflowing the APP_FLASH by 3KB. I tried removing some unnecessary custom images and fonts, but the binary sizes with and without these changes stays the same. Are there any guides here for optimizing App compilation size, i tried looking but didn’t see anything?

armor · October 21, 2019, 1:39pm

A little tip on the flash size - you can go up to 116%/120kB - if it is truly too big then you will get a compile error. It is not recommended and when you are right on the top top limit and the next Device OS needs more space…then you are into refactoring. Have you found the tutorial on code size optimisation? If not, then have a go with that.

TrikkStar · October 21, 2019, 2:21pm

Figures that the “tutorial” would be under Support and not Tutorials tab in the docs, had to use Google to actually find it.

And just for some perspective.
Master

Branch 1

Branch 2

Branch 2 + SDFat

Hopefully I can refactor enough to make everything fit.

armor · October 21, 2019, 2:46pm

Big summer blow-out!

I have a lot of const char array references to file names on SD card - so I chopped down the length of the filename to 3 letters and removed the extension which I add on when I read the files. That gained me sufficient - for the time being!

TrikkStar · October 21, 2019, 3:41pm

I wish I had such a “simple” fix. I just have a bunch of objects and libraries I need to use.
I even tried removing some #define PINFUNCTION PIN# and just instanciated my objects with the Pin#, but that got me nothing.

Also noticed most of the tips were for saving memory but not flash size.

ScruffR · October 21, 2019, 5:16pm

It’s quite difficult to provide specific advice without seeing any of your code.
Just general hints would be to reduce explicit initialisation values for globals and replace repetitive code blocks with loops and/or functions.

BTW, #define PINFUNCTION PIN# won’t use any extra flash space compared just using numbers since the pin macros will be substituted with the respective numbers by the preprocessor prior to compilation anyhow.

armor · October 21, 2019, 5:39pm

The tips cover both flash and RAM usage. If you have RAM free then you can adopt techniques

Using switch() case: in place of if else if saves space. Not 3K in one place but if you have 130K program then I am guessing 20K lines with comments. Probably you need to look at the libraries and make sure you are only compiling what you need.

Are you using SoftAP? That takes a lot of flash. Maybe look at other methods.

Regards your libraries maybe you could implement them yourself but as pared down versions.

It is a lot of effort - nothing is simple. Presumably you were doing continuous integration so that this size should have been evident before!

TrikkStar · October 23, 2019, 3:46pm

Thanks for the advice both of you.

@ScruffR I had though so but this section from the guide confused me

Using const variables is the same as using #define in terms of flash and RAM, however they're better because they have an explicit type.

Though I now understand this to mean variables with actual values and not simple mappings.

@armor I do prefer using switch()case instead of if-then-else trees, unfortunately bools don't work in switch statements. I am likely going to need to re-implement a bunch of libraries myself as I'm not doing SoftAP on the device side (we are hosting a page on our website for that purpose).

Not sure if this is worth breaking into a separate thread or not (as I haven't seen any discussion on here about it), but I'm having a hard time figuring out how to load "images" off a SD card to be used on an ePaper display.

The "images" are currently hard coded into firmware like this:

const unsigned char gImage_plug[72] = { /* 0X00,0X01,0X1A,0X00,0X12,0X00, */
0X00,0X40,0X80,0X00,0X00,0X40,0X80,0X00,0X00,0X40,0X80,0X00,0X00,0X40,0X80,0X00,
0X00,0X40,0X80,0X00,0X03,0XFF,0XF0,0X00,0X03,0XFF,0XF0,0X00,0X03,0X00,0X30,0X00,
0X03,0X80,0X70,0X00,0X01,0X80,0X60,0X00,0X01,0X80,0X60,0X00,0X00,0X80,0X40,0X00,
0X00,0XC0,0XC0,0X00,0X00,0X73,0X80,0X00,0X00,0X3F,0X00,0X00,0X00,0X0C,0X00,0X00,
0X00,0X0C,0X00,0X00,0X00,0X0C,0X00,0X00,};

And the function to display the images takes a

const uint8_t[]

My thought was to store each image in a separate file as just the comma separated hex values on a single line, read that line into a String with SDFat, then cast it to the proper data type

const uint8_t* bmp = reinterpret_cast<const uint8_t*>(bmpStr.c_str());

However this is clearly not working as the image comes out as random pixels.
Any suggestion on how to do this?

ScruffR · October 23, 2019, 3:57pm

If you have a 72 byte file that holds your image, you'd just open the file, read its contents into an automatic/local uint8_t array and pass that to the ePaper library. Once sent the local array can just vanish.

Why would you do that? Just store the raw bytes in the file. No parsing required (a mere cast will never translate a four byte string plus comma into a single byte).

BTW, on the Photon and P1 there is some flash space (16+64KB) set aside to be used for EEPROM. You could facilitate that to store your images in flash too.

TrikkStar · October 23, 2019, 8:30pm

How exactly?
This is the raw output I'm getting from the converter application

const unsigned char gImage_plug[72] = { /* 0X00,0X01,0X1A,0X00,0X12,0X00, */
0X00,0X40,0X80,0X00,0X00,0X40,0X80,0X00,0X00,0X40,0X80,0X00,0X00,0X40,0X80,0X00,
0X00,0X40,0X80,0X00,0X03,0XFF,0XF0,0X00,0X03,0XFF,0XF0,0X00,0X03,0X00,0X30,0X00,
0X03,0X80,0X70,0X00,0X01,0X80,0X60,0X00,0X01,0X80,0X60,0X00,0X00,0X80,0X40,0X00,
0X00,0XC0,0XC0,0X00,0X00,0X73,0X80,0X00,0X00,0X3F,0X00,0X00,0X00,0X0C,0X00,0X00,
0X00,0X0C,0X00,0X00,0X00,0X0C,0X00,0X00,};

I removed the commas leaving just a sequence of hex values, but that doesn't work as each character is read as it's own byte.

This is what I'm doing now (using a different image than linked above)

File img1 = SD.open("Images/st.txt");
  if(img1){
    uint8_t* bmp;
    int index = 0;
    while (img1.available()) {
      bmp[index] = img1.read();
      index ++;
    }
    Serial.println(index);
    epaper.PaintBitmap(0, 25, 295, 58, bmp, false);
    epaper.DispalyFrame();
    img1.close();
  } else{
    Serial.println("error opening st.c");
  }

I think part of my confusion is coming from the fact that, as I understand it, the hex value 0x00 is equivalent to 32 bits or 4 bytes. But each hex entry is being stored as a single uint8_t in the array, which is just a byte. So I'm now trying to read two characters at a time and store that as a single index in the array, but that shouldn't work right?

armor · October 23, 2019, 9:50pm

I think you have got confused here between hexidecimal human readable format and the binary number itself. The SD file could be a binary .bin and not a text file .txt. I use a lot of .bmp format files created straight from Photoshop. Creating image files for epaper displays is possible but a little trickier because of the scan direction used and what 1 means and the packing used.

If each byte was actually stored on the SD file as 0XFF, say then you would need to read in the c-string and convert to a number. [https://stackoverflow.com/questions/29547115/how-to-convert-string-to-hex-value-in-c/29547549] and have a value 11111111.

Not having seen the epaper library you are using I can’t say what the method PaintBitmap() is expecting for bmp but in your code snippet you don’t define the variable bmp as an array to hold the image but merely as a pointer.

ScruffR · October 24, 2019, 3:41am

What converter application? What do you feed into it?

Worst case you could use a hex editor (e.g. Notepad++ with the Hex Edit plugin) and enter the hex values manually.
But looking at your example image code any bitmap editor that can produce 8bit monochrome images should be fine to produce such a file. The only thing you might need to consider when reading the files back is to start off after the BMP header.
e.g. Windows Paint () can store monochrome (1bit pixel depth) BMPs where one byte stands for 8 pixels.

TrikkStar · October 24, 2019, 5:59pm

The method I'm using is a wrapper for this library and is defined below.

Definitions

void EPaper::PaintBitmap(int x, int y, int w, int h, const uint8_t bitmap[], bool inverted)
{
    _whiteout();
    if (inverted){
        _epd.drawInvertedBitmap(x, y, bitmap, w, h, GxEPD_BLACK);
    }
    else{
        _epd.drawBitmap(x, y, bitmap, w, h, GxEPD_BLACK);
    }
}

//Source from Library
// BITMAP / XBITMAP / GRAYSCALE / RGB BITMAP FUNCTIONS ---------------------

/**************************************************************************/
/*!
   @brief      Draw a PROGMEM-resident 1-bit image at the specified (x,y) position, using the specified foreground color (unset bits are transparent).
    @param    x   Top left corner x coordinate
    @param    y   Top left corner y coordinate
    @param    bitmap  byte array with monochrome bitmap
    @param    w   Width of bitmap in pixels
    @param    h   Hieght of bitmap in pixels
    @param    color 16-bit 5-6-5 Color to draw with
*/
/**************************************************************************/
void Adafruit_GFX::drawBitmap(int16_t x, int16_t y,
  const uint8_t bitmap[], int16_t w, int16_t h, uint16_t color) {

    int16_t byteWidth = (w + 7) / 8; // Bitmap scanline pad = whole byte
    uint8_t byte = 0;

    startWrite();
    for(int16_t j=0; j<h; j++, y++) {
        for(int16_t i=0; i<w; i++) {
            if(i & 7) byte <<= 1;
            else      byte   = pgm_read_byte(&bitmap[j * byteWidth + i / 8]);
            if(byte & 0x80) writePixel(x+i, y, color);
        }
    }
    endWrite();
}

I finally got a "working" solution, but the Issue I'm now running into is that even though the image is being loaded and displayed correctly, my device is freezing up afterwards. Data in the file is comma separated two character hex values ("00" instead of "0x00). After displaying the image the device breaths cyan without processing further code for ~17 seconds then goes breaths green for a few seconds before blinking red twice (not in an SOS style error) and then flashing cyan until I reset it. It is also worth noting that nothing is updated on the serial monitor until the red blinks.

Source

setup()
{
//Logic
SDTest();

  if (!Particle.connected()){
    Serial.println("offline");
  }
  else{
    //three subscribes here
    //Cloud variables here

    Particle.publish(***);
    Serial.println("Reporting to *****!");
  }
//More logic that spits out to the serial monitor
}


void SDTest(){
  Serial.print("Initializing SD card...");

  if (!SD.begin(SD_CS_PIN)) {
    Serial.println("initialization failed!");
    return;
  }
  Serial.println("initialization done.");

  File img1 = SD.open("Images/st.txt");
  if(img1){
    uint8_t bmp[2146];
    int index = 0;
    while (img1.available()) {
      char buff[3];
      img1.readBytesUntil(',', buff, 3);
      bmp[index] = *hex_str_to_uint8(buff);
      index ++;
    }
    // Serial.println("I: " + String(index));
    epaper.PaintBitmap(0, 25, 295, 58, bmp, false);
    epaper.DispalyFrame();
  } else{
    Serial.println("error opening st.c");
  }
  img1.close();
}

uint8_t* hex_str_to_uint8(const char* string){
  if (string == NULL){
    return NULL;
  }

  size_t slength = strlen(string);
  if ((slength % 2) != 0){ // must be even
    return NULL;
  }

  size_t dlength = slength / 2;
  uint8_t* data = (uint8_t*)malloc(dlength);
  memset(data, 0, dlength);

  size_t index = 0;
  while (index < slength) {
    char c = string[index];
    int value = 0;

    if (c >= '0' && c <= '9'){
      value = (c - '0');
    }
    else if (c >= 'A' && c <= 'F'){
      value = (10 + (c - 'A'));
    }
    else if (c >= 'a' && c <= 'f'){
      value = (10 + (c - 'a'));
    }
    else{
      return NULL;
    }

    data[(index / 2)] += value << (((index + 1) % 2) * 4);
    index++;
  }
  return data;
}

I'm using Image2Lcd as it's recommended (though I can't find the exact wording atm) on the Waveshare site and feeding it a monochrome .bmp

peekay123 · October 24, 2019, 6:37pm

@TrikkStar, if you writing raw binary bytes to the file, you can read them back into an array directly, no commas necessary. Your system may be crashing due to writing past a byte array’s boundary. Hard to tell without code.

TrikkStar · October 24, 2019, 6:55pm

Not writing raw binary, just copy-pasting the hex values from my const unsigned char[] into a .txt file then formatting it accordingly. You may be right about writing past the array as I’m experiencing the same behavior weather or not I actually update the display. How would I go about testing this, increasing the size of the array by 1 didn’t change anything.

TrikkStar · October 24, 2019, 7:26pm

So it turns out that my crashing issue was caused by not de-allocating the my pointers in a timely manner. Instead of directly passing the underlying value of a pointer, the following fixed my crashing issue. I’m positive there’s better ways to do this, but at least I have a working stating point at this point.

uint8_t bmp[2146];
int index = 0;
while (img1.available()) {
  char buff[3];
  img1.readBytesUntil(',', buff, 3);
  uint8_t* point = hex_str_to_uint8(buff);
  bmp[index] = *point;
  free(point);
  index ++;
}

peekay123 · October 24, 2019, 7:26pm

@TrikkStar, you would need a HEX editor to stuff the raw binary in to an SD file by hand. As for the array, can you post your code where you read the file from the SD so we can make some suggestions?

rickkas7 · October 30, 2019, 2:27pm

There is a new version of the PublishQueueAsyncRK library (0.1.0). This version supports pluggable storage modules, including:

Retained memory
Regular RAM
FRAM (MB85RC256V ferro-electric non-volatile RAM) connected by I2C
SPI NOR Flash (ISSI, Winbond, or Macronix) using SpiffsParticleRK
SPI Flash memory on P1 module
SPI Flash soldered to E Series module (Macronix MX25L8006EM1I-12G)
SD cards using the SdFat library

Topic		Replies	Views
PublishQueueAsyncRK and retain memory size left General argon , photon	1	530	July 16, 2019
PublishQueueAsyncRK Hanging Libraries	4	356	June 3, 2022
Adapting PublishQueueAsyncRK for Flash Memory General electron	10	997	November 22, 2019
PublishQueueAsyncRK repeated old data Libraries	4	376	September 19, 2022
Using PublishQueuePosixRK with QSPI and other FLASH Connectivity boron	4	330	June 16, 2023

PublishQueueAsyncRK and Large Datasets

Related Topics