SleepHelper library (preliminary)

I created a new library with the intention of making it easier to write certain types of applications, primarily on cellular devices and using sleep modes.

Check out the README file in the repository for information about the things the library can take care of for you.

https://github.com/rickkas7/SleepHelper/

Note that this library is currently incomplete and may still contain bugs. I’m releasing it now to get initial feedback and feature suggestions.

There is also full browsable HTML documentation available for the API functions.

11 Likes

@rickkas7 - THANK YOU! I am looking forward to kicking the tires on this library!

Chip

4 Likes

@rickkas7 ,

In perusing the documentation, this looks like a fantastic addition to the Particle platform. I am particularly excited about the future development of cloud-based configuration.

One thing that seems clear is that this library could significantly simplify my code and, I hope, take care of some of the issues that have required workarounds. One of these areas is in battery state of charge as the connection state of the device and the time since sleep can both impact SoC readings.

This brings up the first question / feature request: There has been a significant number of posts about the accuracy of state of charge measurements when devices sleep most of the time.

I was wondering if this library addresses these issues or if we need to add an option for .withShouldConnectMinimumSoC() to look at a minimum battery voltage. Perhaps a .withShouldConnectMinimumVcell()?

I understand this library can be extended as you showed with the wake from button example.

Again, I believe this is a major step forward - thank you!

3 Likes

This is awesome @rickkas7 !

I’d like to throw out a special Thanks for the HTML documentation.

1 Like

Good idea about doing a more elaborate check for SoC. However, since that feature is implemented using the features of the library itself, you could easily implement an arbitrarily complex method of determining if there is sufficient power to connect.

The withShouldConnectMinimumSoC() method really just adds a shouldConnect callback. You could add your own custom shouldConnect callback from your firmware code; you don’t need to modify the library.

    SleepHelper &withShouldConnectMinimumSoC(float minSoC, int conviction = 100) {
        return withShouldConnectFunction([minSoC, conviction](int &connectConviction, int &noConnectConviction) {            
            float soc = System.batteryCharge();
            if (soc >= 0 && soc < minSoC) {
                // Battery is known and too low to connect
                noConnectConviction = conviction;
            }
            return true;
        });
    }
1 Like

@rickkas7 ,

I have been working through your documentation and examples, I appreciate the flexibility of this approach to cover a broad number of use cases.

To get something working now, I plan to add:

  • Support for the AB1805 watchdog
  • Support for publishQueue so I can send data using Webhooks while the server-side code part is being built.

Does this make sense or will these capabilities be added to the library as well?

Thanks, Chip

The AB1805 support should be easy to add in your application, and should not conflict with anything.

There’s part of the integration with PublishQueuePosixRK already. See the withPublishQueuePosixRK() method which hooks the two libraries together. This prevent sleep from occurring before the publish queue is empty.

However, there is one additional problem in that the two libraries don’t rate limit publishes between each other properly. I’ll make the change, but I have not done so yet.

2 Likes

@rickkas7 ,

I am trying to wrap my head around this new approach and I hope you don’t mind all the questions. I can see the power in this approach and how it can make programs easier to write and maintain. My hope is to build a template for applications using this approach and try to keep it as generic as possible for as long as possible. That way, I can invest the time in mastering this new library / approach and then quickly add the specifics of my various projects. My hope is that others will find this useful too.

In this iteration, I have added the AB1805 watchdog timer to the function. I believe this is working correctly but, please let me know if there is a better way to implement this. In addition to the standard commands to setup the watchdog and the command in the main loop, I added these two functions in setup

        .withSleepConfigurationFunction([](SystemSleepConfiguration &sleepConfig, SleepHelper::SleepConfigurationParameters &params) {
            // Stop the watchdog timer while we are sleeping
            ab1805.stopWDT();                                                  // No watchdogs interrupting our slumber
            return true;
        })
        .withWakeFunction([](const SystemSleepResult &sleepResult) {
            // Resume the watchdog timer on wake
            delay(2000);                                                       // Delay so we can capture in serial monitor
            ab1805.resumeWDT();
            return true;
        })

The delay can be removed as the watchdog does seem to be started and stopped as needed:

Serial connection closed.  Attempting to reconnect...
Serial monitor opened successfully:
0000160252 [app.ab1805] INFO: setWDT -1
0000160310 [app.sleep] INFO: stateHandlerStart
0000160310 [app.sleep] INFO: running in no connection mode
0000160311 [app.sleep] INFO: done with no connection mode, preparing to sleep
0000160318 [app.ab1805] INFO: setWDT 0
0000160319 [app.sleep] INFO: stateHandlerSleep
0000160372 [app.sleep] INFO: sleeping for 117 sec adjustmentMs=54

The next step is to add the publisQueue library so I don’t need any server side code running to capture the data. Here is where I have a few questions:

  • I did not see a withPublishQueuePosixRK() in the documentation or in the type ahead
  • Is the rate limiting issue a show stopper for now?
  • Would the publishQueue simply be another “addEvent” to the data capture function?
  • Finally, what if I wanted to capture and report system events such as connection timeouts?

I noticed that you put the connection timeout in you example to 11 minutes. Does that mean that the issue with the Boron not knowing the connection state of the SARA modem has been addressed? Currently, I am using a workaround proposed by support to reset the modem at 3,5 and 7 minutes to reduce the chances of an 11 minute connection period. Would it make sense to set a shorter period in power-sensitive implementations?

Thank you as always for your help,

Chip

1 Like

I’ll have a version of the library with the completed integration between AB1805_RK and PublishQueuePosixRK in the next day or so. It will be a single line to enable, and the documentation for those function will be included in the online docs.

2 Likes

Thank you @rickkas7, I look forward to continuing to learn how to use this powerful new library.

I released version 0.0.2 of the library.

  • Implementation of withPublishQueuePosixRK and withAB1805_RK. Example usage in more-examples/50-publish-queue.

  • Added a new section "Maximum connection time" that describes how to set the value. It can be shorter than 11 minutes if certain other criteria are met.

Some examples use a maximum time to connect:

// EXAMPLE
SleepHelper::instance()
    .withShouldConnectMinimumSoC(9.0)
    .withMaximumTimeToConnect(11min);

// PROTOTYPE
SleepHelper &withMaximumTimeToConnect(std::chrono::milliseconds timeMs); 

If the cloud connection starts but does not successfully complete, this can be safely done at 11 to 12 minutes. The reason is that around 10 minutes, the modem will be powered down, which can clear some temporary conditions in the modem.

However, if you have a battery-sensitive situation (only battery or battery with solar), then you may not want to wait the full 11 minutes. As long as you are using sleep mode, and using it will cellular off, that is sufficient to reset the modem in the same way, so you can use a lower value, possibly as low as 4 minutes. If you are using a 2G/3G device cellular device, you may want to set it a bit longer, 5 to 6 minutes.

If you use this technique to reduce the maximum time to connect, makes sure that you do not set withMinimumCellularOffTime, or set it to a value long enough to assure that the modem will be powered off to make sure it is reset.

  • Added this text to make the difference between addEvent and actual events more clear:

In other words, adding a wake event does not equal a single new publish. Wake events are a fragment of JSON that is added to other fragments of JSON to produce one or more JSON events. This reduces the number of data operations while also freeing you from having to worry about adding so many fragments that you exceed the size of a publish. This is all taken care of by the library.

3 Likes

@rickkas7 , I am struggling a bit with this concept. I can see from the Particle console that the JSON payload is getting these events added and I can see how this would help reduce the number of data operations. Further, the ability to assign priority to the messages makes this a very powerful approach.

But, I don't understand how this big accretion of JSON events gets parsed and sent on to the various back-end services such as Ubidots or AWS IOT, etc. Will we need to stand up and maintain a service to parse these messages? Will there be a service hosted on the Particle platform and served up the same way other integrations are today?

{
  "soc": 79.1,
  "ttc": 112159,
  "rr": 20,
  "eh": [
    {
      "t": 1654112882,
      "c": 22.2
    },
    {
      "t": 1654113002,
      "c": 21.6
    },
    {
      "t": 1654113147,
      "c": 22.1
    }
  ]
}

Also, I tried adding a straight up publish to the data capture function:

        .withDataCaptureFunction([](SleepHelper::AppCallbackState &state) {
            if (Time.isValid()) {
                SleepHelper::instance().addEvent([](JSONWriter &writer) {
                    writer.name("t").value((int) Time.now());
                    writer.name("c").value(readTempC(), 1);
                });
                char dataStr[16];           // While we are here see if I can send a webhook to the queue
                snprintf(dataStr,sizeof(dataStr),"t: %4.2f",readTempC());
                PublishQueuePosix::instance().publish("Test", dataStr, PRIVATE);
                Log.info(dataStr);          // Visibility to the payload in the webhook
            }
            return false;
        })

but this does not work - with this approach is it not possible to send a webhook if we so choose?

here is what I see in the terminal - the dataStr is getting built but not sure what happens to the publish?

Serial connection closed.  Attempting to reconnect...
Serial monitor opened successfully:
0000448235 [app.ab1805] INFO: setWDT -1
0000448291 [app.pubq] TRACE: publishCommon eventName=Test eventData=t: 21.16
0000448292 [app.pubq] TRACE: fileQueueLen=40 ramQueueLen=1 connected=0
0000448398 [app.pubq] TRACE: writeQueueToFiles fileNum=41
0000448399 [app] INFO: t: 21.16
0000448399 [app.sleep] INFO: stateHandlerStart
0000448399 [app.sleep] INFO: running in no connection mode
0000448401 [app.sleep] INFO: done with no connection mode, preparing to sleep
0000448426 [app.sleep] INFO: stateHandlerSleep
0000448426 [app.ab1805] INFO: setWDT 0
0000448473 [app.sleep] INFO: sleeping for 116 sec adjustmentMs=47

If this is as far as we can go until the server-side is ready, please let me know. My hope was to use this Webhook approach as a temporary workaround to keep moving forward with this exploration.

Thanks,

Chip

Well, I guess those PublishQueue publishes did not disappear. After a couple hours, All these were sent in one very long string. After this, the device stayed connected and did not go back to sleep. Here is what I saw in the console and the serial window.

0007802453 [system] INFO: Cloud connected
0007802454 [app.sleep] INFO: connected to cloud in 34400 ms
0007802745 [app.sleep] INFO: removing item from publishData
0007803753 [app.sleep] INFO: removing item from publishData
0007804752 [app.sleep] INFO: removing item from publishData
0007805745 [app.seqfile] TRACE: getFileFromQueue returned 3
0007805761 [app.pubq] TRACE: fileNum=3 size=83
0007805762 [app.pubq] TRACE: readQueueFile 3 event=Test data=t: 23.01
0007805762 [app.pubq] TRACE: publishing file event=Test data=t: 23.01
0007806058 [app.pubq] TRACE: publish success 3
0007806059 [app.seqfile] TRACE: getFileFromQueue returned 3
0007806059 [app.seqfile] TRACE: getFileFromQueue returned 3
0007806116 [app.seqfile] TRACE: removed /usr/pubqueue/00000003
0007806117 [app.pubq] TRACE: removed file 3
0007807117 [app.seqfile] TRACE: getFileFromQueue returned 4
0007807147 [app.pubq] TRACE: fileNum=4 size=83
0007807147 [app.pubq] TRACE: readQueueFile 4 event=Test data=t: 22.61
0007807148 [app.pubq] TRACE: publishing file event=Test data=t: 22.61
0007807422 [app.pubq] TRACE: publish success 4
0007807423 [app.seqfile] TRACE: getFileFromQueue returned 4
0007807423 [app.seqfile] TRACE: getFileFromQueue returned 4
0007807489 [app.seqfile] TRACE: removed /usr/pubqueue/00000004
0007807489 [app.pubq] TRACE: removed file 4
0007808490 [app.seqfile] TRACE: getFileFromQueue returned 5
0007808506 [app.pubq] TRACE: fileNum=5 size=83
0007808506 [app.pubq] TRACE: readQueueFile 5 event=Test data=t: 23.17
0007808507 [app.pubq] TRACE: publishing file event=Test data=t: 23.17
0007808774 [app.pubq] TRACE: publish success 5
0007808775 [app.seqfile] TRACE: getFileFromQueue returned 5
0007808775 [app.seqfile] TRACE: getFileFromQueue returned 5
0007808834 [app.seqfile] TRACE: removed /usr/pubqueue/00000005
0007808834 [app.pubq] TRACE: removed file 5
0007809834 [app.seqfile] TRACE: getFileFromQueue returned 6
0007809863 [app.pubq] TRACE: fileNum=6 size=83
0007809864 [app.pubq] TRACE: readQueueFile 6 event=Test data=t: 22.69
0007809865 [app.pubq] TRACE: publishing file event=Test data=t: 22.69
0007810134 [app.pubq] TRACE: publish success 6
0007810135 [app.seqfile] TRACE: getFileFromQueue returned 6
0007810135 [app.seqfile] TRACE: getFileFromQueue returned 6
0007810201 [app.seqfile] TRACE: removed /usr/pubqueue/00000006
0007810201 [app.pubq] TRACE: removed file 6
0007811201 [app.seqfile] TRACE: getFileFromQueue returned 7
0007811217 [app.pubq] TRACE: fileNum=7 size=83
0007811217 [app.pubq] TRACE: readQueueFile 7 event=Test data=t: 22.85
0007811218 [app.pubq] TRACE: publishing file event=Test data=t: 22.85
0007811434 [app.pubq] TRACE: publish success 7
0007811435 [app.seqfile] TRACE: getFileFromQueue returned 7
0007811435 [app.seqfile] TRACE: getFileFromQueue returned 7
0007811492 [app.seqfile] TRACE: removed /usr/pubqueue/00000007
0007811492 [app.pubq] TRACE: removed file 7
0007812492 [app.seqfile] TRACE: getFileFromQueue returned 8
0007812521 [app.pubq] TRACE: fileNum=8 size=83
0007812521 [app.pubq] TRACE: readQueueFile 8 event=Test data=t: 23.33
0007812522 [app.pubq] TRACE: publishing file event=Test data=t: 23.33
0007812847 [app.pubq] TRACE: publish success 8
0007812848 [app.seqfile] TRACE: getFileFromQueue returned 8
0007812848 [app.seqfile] TRACE: getFileFromQueue returned 8
0007812913 [app.seqfile] TRACE: removed /usr/pubqueue/00000008
0007812913 [app.pubq] TRACE: removed file 8
0007813914 [app.seqfile] TRACE: getFileFromQueue returned 9
0007813929 [app.pubq] TRACE: fileNum=9 size=83
0007813930 [app.pubq] TRACE: readQueueFile 9 event=Test data=t: 22.93
0007813931 [app.pubq] TRACE: publishing file event=Test data=t: 22.93
0007814214 [app.pubq] TRACE: publish success 9
0007814215 [app.seqfile] TRACE: getFileFromQueue returned 9
....
This continued for the next 80 publishes.

And here is what was in the console:

At this point, I am not sure if there is an issue with these two libraries working together or - more likely - I am doing something wrong.

Thanks,

Chip

That’s weird. When I get a chance I’ll test the specific case of adding a publish queue event from the data capture handler. I suspect that the code is mistakenly believing the queue is empty and going right to sleep, but I’m not really sure why that’s happening.

1 Like

@rickkas7 , I am thinking about a way to organize a generic implementation of this code. Here is what I am thinking:

  • main.cpp - this will be the main program and will include standard functionality
  • takeMeasurements.h - Specific to the sensors / measurements for each project
  • interrupts.h - This is where we define and service any specific interrupts
  • current_data.h - an object to hold current data (last measurements, battery level, temperature, etc)
  • system_data.h - an object to hold system configuration data (cell provider, power config, operating hours, measurement frequency, etc)

Not sure about this but it may also make sense to define a header for Particle variables and functions as one additional header.

  • particle_functions.h - where we define the Particle variables and functions if needed.

Would it make sense to organize and maintain things this way? If so, most of the tweaking of the code for any specific application would take place in the header files making it easier to maintain the main.cpp as new features / functions are developed for the sleep helper library.

Thanks,

Chip

2 Likes

@rickkas7 what do you think?

That file organization makes sense. I will test PublishQueuePosixRK from a data capture handler, but I haven’t had a chance to do so yet.

@rickkas7 ,

I have been working on restructuring this example with the intent to make it easier to develop / test an example implementation that should be more easily adapted to specific use cases. I am making progress on this but I am also starting to worry I might be adding not removing complexity. I would appreciate any comments / suggestions you or the community might have.

Please see the current state of this code here: https://github.com/chipmc/SleepHelper-Demo

As the number of files grows, this might be the best way to share.

Here is what I have working so far:

  1. I have a separate header and source file for pin definitions (not sure if this is a good idea or if I have the best implementation). My intent was to make it clear what the pin definitions for a particular implementation are and allow them to be used across the source files as needed.
  2. I broke out the data objects - system (for configuration) and current (for current sensor / device values) so you can see what data will be available and shared across source files
  3. I created a file for Particle functions / variables. I have found these are useful for configuring a device and for checking on variables. These may not be needed based on what your back-end looks like.
  4. Finally, I will put all the code for interacting with the various sensors attached to the device into a “take measurements” file. This file will change based on what is attached to the device.

My next step is to break out the configuration of the scheduled and interrupt configurations and actions into one more file - something like sleep helper config. At that point, there should not be much left in the main source file.

I also need to figure out how to abstract / generalize persistent storage (EEPROM, Flash, FRAM, etc).

Again, I am looking for feedback on whether I am on the right track with this effort or if there could be a better way.

Thanks,

Chip

2 Likes

That sounds like a good plan.

1 Like

@rickkas7 ,

Thank you for the encouragement. I am learning how these functions work and trying to set things up in a way that makes sense. In this iteration, I pulled all the Sleep Helper configuration into a separate library. This makes the main source file very short and generic.

// Include needed Particle / Community libraries
#include "AB1805_RK.h"
#include "PublishQueuePosixRK.h"
#include "SleepHelper.h"
#include "LocalTimeRK.h"
// Include headers that are part of this program's structure
#include "storage_objects.h"                        // Where we define our structures for storing data
#include "device_pinout.h"                          // Where we store the pinout for our device
#include "take_measurements.h"                      // This is the code that collects data from our particular sensor configuration
#include "particle_fn.h"                            // Place where common Particle functions will go
#include "sleep_helper_config.h"                    // This is where we set the parameters for the Sleep Helper library

// Set logging level and Serial port (USB or Serial1)
SerialLogHandler logHandler(LOG_LEVEL_INFO, {       // Changed to USB log handler from Serial1
	{ "app.pubq", LOG_LEVEL_TRACE },                // Add additional logging for PublishQueuePosixRK
	{ "app.seqfile", LOG_LEVEL_TRACE }              // And the underlying sequential file library used by PublishQueuePosixRK
});                        

// Set the system modes
SYSTEM_THREAD(ENABLED);
SYSTEM_MODE(SEMI_AUTOMATIC);
STARTUP(System.enableFeature(FEATURE_RESET_INFO));  // So we know why the device reset

// Instantiate services and objects
AB1805 ab1805(Wire);                                // Rickkas' RTC / Watchdog library
struct systemStatus_structure sysStatus;            // See structure definition in storage_objects.h
struct current_structure current;                   

// Variables
char tempString[16];

// Support for Particle Products (changes coming in 4.x - https://docs.particle.io/cards/firmware/macros/product_id/)
PRODUCT_ID(PLATFORM_ID);                            // Device needs to be added to product ahead of time.  Remove once we go to deviceOS@4.x
PRODUCT_VERSION(0);
char currentPointRelease[6] ="0.05";


void setup() {

    Particle.variable("tempC", tempString);

    Particle.function("Set Mode", setLowPowerMode);
    Particle.function("Set Wake Time", setWakeTime);
    Particle.function("Set Sleep Time", setSleepTime);

    // Initialize AB1805 Watchdog and RTC
    {
        ab1805.setup();

        // Reset the AB1805 configuration to default values
        ab1805.resetConfig();

        // Enable watchdog
        ab1805.setWDT(AB1805::WATCHDOG_MAX_SECONDS);
    }

    // Initialize PublishQueuePosixRK
	PublishQueuePosix::instance().setup();

    // Configure and initialize the Sleep Helper function
    sleepHelperConfig();                                 // This is the function call to configure the sleep helper parameters

    SleepHelper::instance().setup();                    // This puts these parameters into action
}

void loop() {
    SleepHelper::instance().loop();

    ab1805.loop();
    
    PublishQueuePosix::instance().loop();
}

Again, if anyone has a suggestion on how to do this in a smarter or more intuitive way, I am all ears.

At this point, there are just two items that need attention:

  1. I need to figure out how to add persistent storage
  2. I would ask you to take a look at PublishQueuePosixRK when you get a chance.

Thank, I will start testing with some local devices to see how they behave in the field.

Thanks,

Chip

1 Like