Daylight Savings Time ... Solution?

Bear · September 3, 2017, 7:54pm

I started work on a potential solution for automating DST on my Photon’s last week, and I’ve gotten far enough to know that all of the puzzle pieces exist. Before I go any further I want to do a sanity check to see how this solution sounds to others …

Basic Concepts.

Store the IANA defined time zone ID (aka Olson Name) in the Photon’s eprom. (Examples of these IDs include:“America/New_York”, “Asia/Katmandu”, “Austrailia/Sydney”, “Europe/Helsinki”, … etc.) IANA periodically publishes the “authoritative” time zone database used by OS and Compiler vendors, and the IANA defined zone ID unlocks the DST rules in that database.
Maintain a ".JSON file (for each time zone) on a server that the Photon can download. This file would contain the standard offset, current zone offset, DST offset, DST (true/false), and DST transition information for all time zones. The Transition information would include the transition date in epoch(seconds) and ISO8601 format along with the new time zone offset, DST offset, and DST(true/false) settings.
On boot-up, Photon firmware would execute a tzQuery() function get the JSON file for the time-zone recorded in the eprom (or UTC by default). The function would then:
a) parse the JSON file
b) update the Photon’s current offsets and DST settings
c) store (or update) the settings and transition information in eprom
In the firmware “loop”, a function would be executed to perform the DST transition when current-time >= the DST transition-time. In normal operation, this would add very little overhead to the loop. The function would merely exit after determining that the transition time is still in the future. When the transition time arrives, the function would update the Photon’s offset and DST settings, and it would move the transition settings to current settings in eprom. To avoid all Photons hitting the server at the same time, the function would also select a random time to retrieve fresh JSON data.

To Date

The biggest question I had going into this was: “How do we generate the JSON data?”. It turns out that Java 8 introduced a lot of new functionality related to zone rules. Armed with this functionality, I was able to generate the JSON data in 60 lines of loosely packed code. At present all of the data resides in a single file, but it will be easy to generate separate files after I/we determine the best download method … HTTP?.

Here is a subset of the JSON data I was able to generate:

America/Kentucky/Louisville = {"StdOffset":-5.0,"CurrentOffset":-4.0,"DST":true,"NextTransition":{,"EpochSeconds":1509861600,"ISO8601":"2017-11-05T01:00-05:00[America/Kentucky/Louisville],"Offset":-5.0,"DST":false"}}
America/Kentucky/Monticello = {"StdOffset":-5.0,"CurrentOffset":-4.0,"DST":true,"NextTransition":{,"EpochSeconds":1509861600,"ISO8601":"2017-11-05T01:00-05:00[America/Kentucky/Monticello],"Offset":-5.0,"DST":false"}}
America/Knox_IN = {"StdOffset":-6.0,"CurrentOffset":-5.0,"DST":true,"NextTransition":{,"EpochSeconds":1509865200,"ISO8601":"2017-11-05T01:00-06:00[America/Knox_IN],"Offset":-6.0,"DST":false"}}
America/Kralendijk = {"StdOffset":-4.0,"CurrentOffset":-4.0,"DST":false}}
America/La_Paz = {"StdOffset":-4.0,"CurrentOffset":-4.0,"DST":false}}
America/Lima = {"StdOffset":-5.0,"CurrentOffset":-5.0,"DST":false}}
America/Los_Angeles = {"StdOffset":-8.0,"CurrentOffset":-7.0,"DST":true,"NextTransition":{,"EpochSeconds":1509872400,"ISO8601":"2017-11-05T01:00-08:00[America/Los_Angeles],"Offset":-8.0,"DST":false"}}
America/Louisville = {"StdOffset":-5.0,"CurrentOffset":-4.0,"DST":true,"NextTransition":{,"EpochSeconds":1509861600,"ISO8601":"2017-11-05T01:00-05:00[America/Louisville],"Offset":-5.0,"DST":false"}}
America/Lower_Princes = {"StdOffset":-4.0,"CurrentOffset":-4.0,"DST":false}}
America/Maceio = {"StdOffset":-3.0,"CurrentOffset":-3.0,"DST":false}}
America/Managua = {"StdOffset":-6.0,"CurrentOffset":-6.0,"DST":false}}
America/Manaus = {"StdOffset":-4.0,"CurrentOffset":-4.0,"DST":false}}
America/Marigot = {"StdOffset":-4.0,"CurrentOffset":-4.0,"DST":false}}
America/Martinique = {"StdOffset":-4.0,"CurrentOffset":-4.0,"DST":false}}
America/Matamoros = {"StdOffset":-6.0,"CurrentOffset":-5.0,"DST":true,"NextTransition":{,"EpochSeconds":1509865200,"ISO8601":"2017-11-05T01:00-06:00[America/Matamoros],"Offset":-6.0,"DST":false"}}
America/Mazatlan = {"StdOffset":-7.0,"CurrentOffset":-6.0,"DST":true,"NextTransition":{,"EpochSeconds":1509264000,"ISO8601":"2017-10-29T01:00-07:00[America/Mazatlan],"Offset":-7.0,"DST":false"}}
America/Mendoza = {"StdOffset":-3.0,"CurrentOffset":-3.0,"DST":false}}
America/Menominee = {"StdOffset":-6.0,"CurrentOffset":-5.0,"DST":true,"NextTransition":{,"EpochSeconds":1509865200,"ISO8601":"2017-11-05T01:00-06:00[America/Menominee],"Offset":-6.0,"DST":false"}}
America/Merida = {"StdOffset":-6.0,"CurrentOffset":-5.0,"DST":true,"NextTransition":{,"EpochSeconds":1509260400,"ISO8601":"2017-10-29T01:00-06:00[America/Merida],"Offset":-6.0,"DST":false"}}
America/Metlakatla = {"StdOffset":-9.0,"CurrentOffset":-8.0,"DST":true,"NextTransition":{,"EpochSeconds":1509876000,"ISO8601":"2017-11-05T01:00-09:00[America/Metlakatla],"Offset":-9.0,"DST":false"}}
America/Mexico_City = {"StdOffset":-6.0,"CurrentOffset":-5.0,"DST":true,"NextTransition":{,"EpochSeconds":1509260400,"ISO8601":"2017-10-29T01:00-06:00[America/Mexico_City],"Offset":-6.0,"DST":false"}}
America/Miquelon = {"StdOffset":-3.0,"CurrentOffset":-2.0,"DST":true,"NextTransition":{,"EpochSeconds":1509854400,"ISO8601":"2017-11-05T01:00-03:00[America/Miquelon],"Offset":-3.0,"DST":false"}}
America/Moncton = {"StdOffset":-4.0,"CurrentOffset":-3.0,"DST":true,"NextTransition":{,"EpochSeconds":1509858000,"ISO8601":"2017-11-05T01:00-04:00[America/Moncton],"Offset":-4.0,"DST":false"}}
America/Monterrey = {"StdOffset":-6.0,"CurrentOffset":-5.0,"DST":true,"NextTransition":{,"EpochSeconds":1509260400,"ISO8601":"2017-10-29T01:00-06:00[America/Monterrey],"Offset":-6.0,"DST":false"}}
America/Montevideo = {"StdOffset":-3.0,"CurrentOffset":-3.0,"DST":false}}
America/Montreal = {"StdOffset":-5.0,"CurrentOffset":-4.0,"DST":true,"NextTransition":{,"EpochSeconds":1509861600,"ISO8601":"2017-11-05T01:00-05:00[America/Montreal],"Offset":-5.0,"DST":false"}}
America/Montserrat = {"StdOffset":-4.0,"CurrentOffset":-4.0,"DST":false}}
America/Nassau = {"StdOffset":-5.0,"CurrentOffset":-4.0,"DST":true,"NextTransition":{,"EpochSeconds":1509861600,"ISO8601":"2017-11-05T01:00-05:00[America/Nassau],"Offset":-5.0,"DST":false"}}
America/New_York = {"StdOffset":-5.0,"CurrentOffset":-4.0,"DST":true,"NextTransition":{,"EpochSeconds":1509861600,"ISO8601":"2017-11-05T01:00-05:00[America/New_York],"Offset":-5.0,"DST":false"}}
America/Nipigon = {"StdOffset":-5.0,"CurrentOffset":-4.0,"DST":true,"NextTransition":{,"EpochSeconds":1509861600,"ISO8601":"2017-11-05T01:00-05:00[America/Nipigon],"Offset":-5.0,"DST":false"}}
America/Nome = {"StdOffset":-9.0,"CurrentOffset":-8.0,"DST":true,"NextTransition":{,"EpochSeconds":1509876000,"ISO8601":"2017-11-05T01:00-09:00[America/Nome],"Offset":-9.0,"DST":false"}}
America/Noronha = {"StdOffset":-2.0,"CurrentOffset":-2.0,"DST":false}}
America/North_Dakota/Beulah = {"StdOffset":-6.0,"CurrentOffset":-5.0,"DST":true,"NextTransition":{,"EpochSeconds":1509865200,"ISO8601":"2017-11-05T01:00-06:00[America/North_Dakota/Beulah],"Offset":-6.0,"DST":false"}}
America/North_Dakota/Center = {"StdOffset":-6.0,"CurrentOffset":-5.0,"DST":true,"NextTransition":{,"EpochSeconds":1509865200,"ISO8601":"2017-11-05T01:00-06:00[America/North_Dakota/Center],"Offset":-6.0,"DST":false"}}
America/North_Dakota/New_Salem = {"StdOffset":-6.0,"CurrentOffset":-5.0,"DST":true,"NextTransition":{,"EpochSeconds":1509865200,"ISO8601":"2017-11-05T01:00-06:00[America/North_Dakota/New_Salem],"Offset":-6.0,"DST":false"}}
America/Ojinaga = {"StdOffset":-7.0,"CurrentOffset":-6.0,"DST":true,"NextTransition":{,"EpochSeconds":1509868800,"ISO8601":"2017-11-05T01:00-07:00[America/Ojinaga],"Offset":-7.0,"DST":false"}}
America/Panama = {"StdOffset":-5.0,"CurrentOffset":-5.0,"DST":false}}
America/Pangnirtung = {"StdOffset":-5.0,"CurrentOffset":-4.0,"DST":true,"NextTransition":{,"EpochSeconds":1509861600,"ISO8601":"2017-11-05T01:00-05:00[America/Pangnirtung],"Offset":-5.0,"DST":false"}}
America/Paramaribo = {"StdOffset":-3.0,"CurrentOffset":-3.0,"DST":false}}
America/Phoenix = {"StdOffset":-7.0,"CurrentOffset":-7.0,"DST":false}}
America/Port-au-Prince = {"StdOffset":-5.0,"CurrentOffset":-4.0,"DST":true,"NextTransition":{,"EpochSeconds":1509861600,"ISO8601":"2017-11-05T01:00-05:00[America/Port-au-Prince],"Offset":-5.0,"DST":false"}}
America/Port_of_Spain = {"StdOffset":-4.0,"CurrentOffset":-4.0,"DST":false}}
America/Porto_Acre = {"StdOffset":-5.0,"CurrentOffset":-5.0,"DST":false}}
America/Porto_Velho = {"StdOffset":-4.0,"CurrentOffset":-4.0,"DST":false}}
America/Puerto_Rico = {"StdOffset":-4.0,"CurrentOffset":-4.0,"DST":false}}
America/Punta_Arenas = {"StdOffset":-3.0,"CurrentOffset":-3.0,"DST":false}}
America/Rainy_River = {"StdOffset":-6.0,"CurrentOffset":-5.0,"DST":true,"NextTransition":{,"EpochSeconds":1509865200,"ISO8601":"2017-11-05T01:00-06:00[America/Rainy_River],"Offset":-6.0,"DST":false"}}
Europe/Isle_of_Man = {"StdOffset":0.0,"CurrentOffset":1.0,"DST":true,"NextTransition":{,"EpochSeconds":1509238800,"ISO8601":"2017-10-29T01:00Z[Europe/Isle_of_Man],"Offset":0.0,"DST":false"}}
Europe/Istanbul = {"StdOffset":3.0,"CurrentOffset":3.0,"DST":false}}
Europe/Jersey = {"StdOffset":0.0,"CurrentOffset":1.0,"DST":true,"NextTransition":{,"EpochSeconds":1509238800,"ISO8601":"2017-10-29T01:00Z[Europe/Jersey],"Offset":0.0,"DST":false"}}
Europe/Kaliningrad = {"StdOffset":2.0,"CurrentOffset":2.0,"DST":false}}
Europe/Kiev = {"StdOffset":2.0,"CurrentOffset":3.0,"DST":true,"NextTransition":{,"EpochSeconds":1509238800,"ISO8601":"2017-10-29T03:00+02:00[Europe/Kiev],"Offset":2.0,"DST":false"}}
Europe/Kirov = {"StdOffset":3.0,"CurrentOffset":3.0,"DST":false}}
Europe/Lisbon = {"StdOffset":0.0,"CurrentOffset":1.0,"DST":true,"NextTransition":{,"EpochSeconds":1509238800,"ISO8601":"2017-10-29T01:00Z[Europe/Lisbon],"Offset":0.0,"DST":false"}}
Europe/Ljubljana = {"StdOffset":1.0,"CurrentOffset":2.0,"DST":true,"NextTransition":{,"EpochSeconds":1509238800,"ISO8601":"2017-10-29T02:00+01:00[Europe/Ljubljana],"Offset":1.0,"DST":false"}}
Europe/London = {"StdOffset":0.0,"CurrentOffset":1.0,"DST":true,"NextTransition":{,"EpochSeconds":1509238800,"ISO8601":"2017-10-29T01:00Z[Europe/London],"Offset":0.0,"DST":false"}}
Europe/Luxembourg = {"StdOffset":1.0,"CurrentOffset":2.0,"DST":true,"NextTransition":{,"EpochSeconds":1509238800,"ISO8601":"2017-10-29T02:00+01:00[Europe/Luxembourg],"Offset":1.0,"DST":false"}}
Europe/Madrid = {"StdOffset":1.0,"CurrentOffset":2.0,"DST":true,"NextTransition":{,"EpochSeconds":1509238800,"ISO8601":"2017-10-29T02:00+01:00[Europe/Madrid],"Offset":1.0,"DST":false"}}
Europe/Malta = {"StdOffset":1.0,"CurrentOffset":2.0,"DST":true,"NextTransition":{,"EpochSeconds":1509238800,"ISO8601":"2017-10-29T02:00+01:00[Europe/Malta],"Offset":1.0,"DST":false"}}
Europe/Mariehamn = {"StdOffset":2.0,"CurrentOffset":3.0,"DST":true,"NextTransition":{,"EpochSeconds":1509238800,"ISO8601":"2017-10-29T03:00+02:00[Europe/Mariehamn],"Offset":2.0,"DST":false"}}
Europe/Minsk = {"StdOffset":3.0,"CurrentOffset":3.0,"DST":false}}
Europe/Monaco = {"StdOffset":1.0,"CurrentOffset":2.0,"DST":true,"NextTransition":{,"EpochSeconds":1509238800,"ISO8601":"2017-10-29T02:00+01:00[Europe/Monaco],"Offset":1.0,"DST":false"}}
Europe/Moscow = {"StdOffset":3.0,"CurrentOffset":3.0,"DST":false}}

KyleG · September 5, 2017, 2:21am

Let me ping someone that might be able to help, @rickkas7 are you able to assist?

Kyle

Bear · September 10, 2017, 2:26am

Progress:

I set up an Ubuntu server that runs on VirtualBox. It hosts an HTTP server.
I modified my Java program to create a JSON file for each time zone. It writes directly to the HTTP server.
I developed a “tzLib” library that exposes two main functions called tzSetup() and tzLoop().

tzSetup()

Is designed to run in the firmware’s Setup() section.
Reads EEProm memory to determine the time zone ID.
Downloads and parses the JSON file for that time zone ID.
Writes offset and DST information to EEProm so the Photon has everything it needs to survive reboots and network outages.
Updates the Particle’s local time zone settings.
I used Serial.println() to generate the following output from my test firmware after tzSetup() ran.

Local TimeZone = US/Central
Standard Offset = -6.000000 hours
Current Offset = -5.000000 hours
Current DST settings: DST = True, DST Offset = 1.000000 hours
Next DST Transition @ 1509865200(Epoch Seconds - UTC)
Next DST transition (local time): Sun Nov  5 02:00:00 2017

tzLoop()

Is designed to run within the firmware’s Loop().
Automatically updates the Photon’s local time zone settings when the DST transition time arrives.
Periodically contacts the HTTP server to maintain the latest timezone information in EEPROM.

Status
This code meets my needs, and I look forward to the next DST transition. A number of things would need to be done make this into a “real” DST solution.

We’d need to maintain the JSON files on an HTTP server that is available on the internet, and that has a fixed DNS name and file path so we can code it into the library.
We’d need to figure out how to initially configure devices with their time zone ID. There are a variety of time zone selectors around. Some are map based, which could be very user friendly. I wrote tzSetup() so it will accept a time zone ID as an optional argument, and tzSetup() can be rerun at any time. It should therefore be easy to change time zone IDs via a call from loop() or from a Particle.function.
We’d need to get someone with stronger c++ skills to take the firmware library to the next level.

Bear · September 23, 2017, 4:55pm

I’m working to assemble what I need to make this solution available to others …

I’ve created a GitHub project (my first). https://github.com/rwpalmer/tzLib . I’d appreciate anyone’s comments and whatever guidance they can offer. The project includes the firmware library and a Java component. I’m not quite sure how to structure that. I’d rather not have two projects, because it is a single solution.
I contacted Particle to see if they would be willing to host the HTTP data that this solution requires. They suggested that I work through the “Community” to find a host. In my view, anything goes during development and test, but we need a permanent URL with a reliable server behind it before people start building this solution into their firmware. Again, any suggestions would be appreciated.
For testing, I loaded Oracle’s"Virtual Box" software on my Windows laptop. This let me create a virtual machine that runs an Apache HTTP server on top of Ubuntu Server. _If anyone want’s to help test this library, I can provide an OVA file with an image of my HTTP server. This can then be imported into Virtual Box or another hypervisor like VMware, KVM, or HyperV. A couple of minor tweaks would need to be made, but they should be painless.

kasper · September 24, 2017, 9:22am

Hi interesting topic. I know DST is not really good implemented (see this discussion and this one with input from different people @ScruffR @wgbartley @pomplesiegel @mebrunet as well).

I like the concepts you present, although I’m wondering why you use the JAVA and JSON route?

Is it not easier to set up a PHP script that gives you the data when you do a POST/GET with the Olson name? PHP already has implemented timezone information.
Besides that I would rather use CSV. I know JSON is nicer to read, however if you know the data structure CSV would be easier and faster to parse on a microcontroller. Nonetheless generating 600 JSON files seems a bit an overkill.

It seems @mebrunet already made a web hook implementation that get the next transition. I don’t know if there is source code available to run this on an own server?

Bear · September 24, 2017, 5:06pm

Thanks for responding !

My choice of Java over PHP was simply because I have experience with Java and I had read about the ZoneRules library that Oracle released with Java 8. I have no experience with PHP, and I was unaware of the PHP option. PHP may well be the better choice.

My Java program runs runs daily to assure that the prestaged JSON files reflect current data. I believe the PHP option would calculate the latest data with each query. PHP would therefore provide up to the second data, and eliminate the need for files to be stored on the HTTP server … right? Would you be willing to help me explore this option?

tzLib currently uses the HttpClient library to “get” the prestaged JSON file for the devices timezone with the Olson name. I’d prefer not to depend on another library, but this library appears to be reliable and efficient … so I opted to use it. Would we do the same if we used PHP?

My choice for JSON over CSV came out of a personal preference for a named value format when exchanging data between loosely coupled programs. I also thought JSON was best since the content was going to be on a HTTP server, where it could be easily be parsed with jquery, should anyone have another use for the data. If we switch to the PHP approach, CSV may well be the better option.

This is how I parse the JSON in the current version of the library.

        // Parse the JSON Data in a non-elegant, but economical manner. 
        char* pch;
        bool parsingError = false;
        pch = strstr(http.body,"\"StdOffset\":");
        if (pch != NULL) { 
            web.stdOffset = atoi(pch+12);
        } else {
            parsingError = true;
        }
        pch = strstr(http.body,"\"CurrentOffset\":");
        if (pch != NULL) { 
            web.currentOffset = atoi(pch+16);
        } else {
            parsingError = true;
        }
        pch = strstr(http.body,"\"EpochSeconds\":");
        if (pch != NULL) { 
            web.transitionTime = atol(pch+15);
        }
        pch = strstr(http.body,"\"Offset\":");
        if (pch != NULL) { 
            web.transitionOffset = atoi(pch+9);
        }

kasper · September 25, 2017, 7:16am

Clear. Yes, PHP will directly respond with JSON data for example.

I hope @mebrunet would like to chime in, because he already seems to have something like this. So that would save us from inventing the wheel again.

daneboomer · July 6, 2018, 9:49am

Have Particle offered to host this @Bear? Seems odd they want the community to host when you’ve actually done them (and the community) a big favour here. Or have you come up with another solution?

kasper · July 6, 2018, 10:12am

I did a PHP implementation with CSV so you can call that with a webhook. Although I’ve lost my particle code. Maybe it comes in handy. The difference with the solution of @Bear is that he uses JSON.

gist.github.com

https://gist.github.com/kasperkamperman/1ca9932d05fbcebe429af7ececcdfc41

getTzDstInfoCSV.php

<?php

    ini_set('display_errors', '1');
    ini_set('error_reporting', E_ALL);

    header('Content-Type: text/csv');

    $isDebug = false;

    if(isset($_POST['debug'])) {

This file has been truncated. show original

index.htm

<!DOCTYPE html>
<html>
    <head>
        <title>test timezone stuff</title>
    </head>
    <body>
    <h2>Select Olson timezone</h2>
    <p>
    <form method="post" action="getTzDstInfoCSV.php">
        <select name="timeZoneID">

This file has been truncated. show original

Parsing could be something like (other function, so not for this data):

int fCallback(String cmd) {

  if(cmd != NULL) {
    har inputCharArray[16];
  	  cmd.toCharArray(inputCharArray,16);

  	    // http://www.cplusplus.com/reference/cstring/strtok/
  		char *token = strtok(inputCharArray,",");

  	  // atoi converts a string to int
  	 // constrain makes sure the value stays in the right range
  	  // strtok breaks string in apart based on the set delimiter (,)
  	  int16_t hue    = constrain(atoi(token),-1,23);
  	  token      = strtok(NULL,",");

    return 1;
  }

  // everything failed
  return -1;
}

Bear · July 6, 2018, 3:34pm

I never found someone to host the HTTP side of the TzLib solution for public use … but at least one organization has created their own server. Last I heard, they were planning to use the solution internally.

A couple of months ago, I released a library called TzCfg. That library uses other public sources and offers timezone and DST configuration based on the device’s IP address, GPS coordinates, or a user-provided time zone ID.

TzCfg is a listed library on the Particle site… so it can be easily added to Particle projects. Documentation is available on GitHub: TzCfg

Topic		Replies	Views
Day Light Saving Time (DST) Library-Photon Firmware	5	2471	March 26, 2018
Time.zone() does not work as documented for Photon? Firmware	3	2381	December 2, 2016
Documentation for Time() functions Getting Started	5	10850	October 29, 2015
Setting a custom device time, without having to change SYSTEM_MODE Firmware	2	743	February 24, 2019
Setting specific event time alarms over 24hour clock - Photon General	1	2383	June 2, 2016

Daylight Savings Time ... Solution?

Related Topics