Semi-Automatic Mode Question

Very useful, but as you said, paraphrasing: The situation is in flux. E.g. your advice to call SPARK_WLAN_Loop() [which I snarkily :smile: point out remains undocumented] is I think now deprecated, and we’re supposed to call Spark.process(). Or are we?

I think also there are WiFi housekeeping issues which need to be attended to even if not connected to the Cloud, and so I think Spark.process() [or the other function] need to be called anyway. The reassurance I seek in that regard is that Spark.process() won’t block if not connected to the Cloud.

That would allow us to write apps which are agnostic to the connection status of the Spark Core to the Cloud. One could write user code which you knew would not block, would use the Cloud when it was available, and not use it when not. And same for the WiFi.

But I think the advice above might be summarised as “quickly test Spark.connected() before the user code blocks.” Or, as may well be the case, am I missing something?

1 Like

@BDub

Since the SPARK_WLAN_Loop() function is not documented I made it like that:

if (!Spark.connected()) { 
    Spark.connect();                            // Establish connection to the cloud
    delay(10000);                                // Give the connection some time to settle to connectivity
        while (!Spark.connected()) {
            RGB.color(255,0,0);                 // Set Core LED to false (red) to get visual feedback that we currently have no connectivity
            Spark.connect();                    // While we are in the loop try to connect (in case internet  is back)
            digole.clearScreen();               // User code not running, to not waste energy clear the display
            delay(60000);                       // This is a worst-case scenario, let´s pause the game for 60 seconds
        }
} else { 
// running threads
}

Tested - works :slight_smile: and delays() in that example are ok for me. Because this code will only be executed if we have no connection :wink:

I agree with psb777. The current implementation of the MANUAL mode makes it less useful than it first appears to be.

My experience: I’m trying to use the Spark Core to build a product to monitor and control electrical devices in homes. Recently my broadband connection at home has become poor with drop-outs and congestion. I’m planning on changing broadband provider, but before I do that I’m treating this as an opportunity to understand how the Spark Core works and whether I can make my product robust in the face of intermittent connectivity outages (which some of my future customers will no doubt have).

The current answer (as per discussions above) appears to be that today I can’t make my product robust. For example, sometime after the Spark Core has successfully connected to the cloud, the connection to the cloud fails, and I see my Spark Core go into say 30 minutes of repeatedly attempting to connect to the Wifi and then Spark Cloud. Each iteration takes around 15 seconds During this time I believe that my user code is not running, and so electrical devices in my home are not monitored and controlled.

My product can tolerate say 15 seconds outage on occasion but it can’t safely tolerate outages of minutes. As I understand the MANUAL mode, I can’t avoid these long occasional outages.

What I need is a new VERYMANUAL mode where I can specify the maximum number of sequential re-connection attempts by the Spark Core to Wifi and the Spark Cloud before execution is returned to the user-loop. In my case I would specify the maximum number of sequential re-connection attempts as 1, so I could be sure my user-loop would get executed often. When my user-loop was executed, it would run my standard monitoring and control processing, then note that the Spark Core was not connected to the cloud, and ask Spark to try to reconnect to the cloud (again attempting to reconnect just once).

In my mind this would give the Spark Core a much more useable balance between executing the user-loop and providing access to the cloud.

2 Likes

One of the replies I had to the suggestion that MANUAL mode might work as @philipq and I each seem to wish it to work is that this is impossible because some calls to the TI CC3000 chip are blocking.

I (naively?) think whereas that may be true (I don’t know) that this is of no consequence: The Core knows if the Cloud is connected, when it becomes disconnected the firmware must just make no blocking calls to the Cloud. I accept that if the WiFi goes down perhaps the Core may block, but the Cloud is just (I think) a TCP connection? It isn’t the WiFi which is down, it is simply the TCP-over-WiFi-connection-to-the-Cloud which does not work. What the two of us want for each of our own purposes is simply to have a mode which is connected to the Cloud when it can be, but is not when it can’t be (ISP problem, problem at Spark, etc).

Please give us a mode which is WiFi-up and Cloud-don’t-care. Then we can write code which reports back to the Cloud or takes instructions from the Cloud but only when the cloud is up, AND which can continue to do the essential local tasks it is programmed to do.

Or tell us how to code to have the same effect. :smile: Already my code looks like as follows. (Ordinarily the Cloud is not connected but I hold a pin high when I want to connect to the Cloud for re-flashing. I would like to routinely report to the Cloud but I cannot because then, same as @philipq, sometime the Core stops working because whereas the LAN is up the WAN is down.).

if( Spark.connected()) Spark.process();
if( Spark.connected()) Spark.publish(...);

but there is still the opportunity to block as the Cloud can evaporate between test and action. It would be nice if these Spark.* calls and other underlying functionality could just silently fail (providing a return code to be tested if required) if the Cloud is unavailable.

I am sure this must be a common user requirement for many types of application.

2 Likes

Well I’m at a bit of a crossroads in terms of using Spark Cores in the development of my product.

In https://community.spark.io/t/semi-automatic-mode-question/7855 a Spark engineer says “If the Cloud disconnects unintentionally, the Core will continue to try to reconnect to the Cloud and will block execution of user code when attempting to connect to the Cloud.” Unintentional Cloud disconnections will occur sooner or later - possibly in the Spark Cloud but also for reasons outside Spark’s control (e.g. my ISP DSLAM or backhaul fails, router faults etc). The outage could last hours or days. During that period the Spark Core imitates a brick.

So I can’t rely on the Spark Core running my user code at all times. So I can’t safely rely on the Spark Core to control electrical devices. The possibility of a sustained outage is a massive risk to the credibility of my product and hence my company.

In https://community.spark.io/t/announcement-introducing-the-photon/8005/88 there is a suggestion that this issue will be resolved by Photon in March 2015. If confirmed, that’s positive, but 3 to 4 months delay is just too slow.

I need to get samples of my product into live environments in mid January to collect performance data. I don’t see a way to safely achieve this using the Spark Core.

I’m coming internal pressure to offload functionality from the Spark Core by adding an Ardunio to the product. This would add cost and complexity to the product and lengthen the development lifecycle. Frankly I don’t want to do this but may be forced to.

Can @zach or @Dave comment on this? Do I actually have any (timely and safe) options? Thanks in advance.

2 Likes

I would like to use the Cloud but I have been forced to stop using it at all. For the reasons you give here. And for the reasons I give elsewhere. There is much overlap in the reasons but essentially for both of us it is this: The Core cannot block just because the ISP has a hiccup. Even if I build MyOwnCloud blocking cannot be allowed.

It does seem to me the issue could be addressed, but I’ve been told by the powers that be that the focus is on the Photon, and the Core will not receive much attention at least for the time being.

The Photon will not have this issue. If the Wifi drops the User Code will continue to run. Yay!

3 Likes

It does look positive for the Photon resolving this issue. But I haven’t seen a 100% commitment to this by Spark. Best commitment I have seen from Spark is here: https://community.spark.io/t/announcement-introducing-the-photon/8005/88 where you say “Does this mean that the Usercode will run even if the Cloud disconnects and your in AUTOMATIC_MODE?? If so I’ll be super happy” and @Zach replies saying “Almost definitely, because we’ll actually be running multiple threads with FreeRTOS, instead of just faking it like we do now.” which is good but not a 100% commitment in my book.

When it gets fixed that will be great, but the fix is not 100% committed to and is 3 to 4 months away. Which is a long time in the IoT space.

I’m starting to think that the lack of comment by @zach and @Dave on this question is an answer in itself.

1 Like

Hi @philipq, don’t read too much into our silence; we’re just very busy :smile:

@philipq, the CC3000 has a blocking driver. It is not possible, without implementing a true RTOS (for which there is not really enough available memory along with the rest of the stack), to make a connect() call and run your own code on the Core simultaneously.

It is possible to change the logic of how many times the Core tries to connect and, when it fails to connect, what it does. If you don’t want the Core to try connecting forever, I’d encourage you to dig into the firmware libraries and make it so that the Core will attempt to connect once, then run your code for a little while, then try to connect again, then run your code for a while, etc. This does require modifying the underlying firmware libraries, as this isn’t a behavior we currently support.

The Photon will almost definitely resolve this issue completely. I say almost simply because we haven’t yet written the software, and I don’t want to make promises I can’t keep. But the architecture of the Photon includes an RTOS to manage user code and connectivity code simultaneously, so we should no longer see any issues with this on the new hardware.

5 Likes

Just to add to this, there have also been a variety of solutions involving watchdogs. You can absolutely, with a combination of watchdog timers, manual connection management, and so on, build something that doesn’t hang, and keeps operating. That being said, network operations are always going to have delays and drops, so if you need absolute realtime, I’ve seen lots of projects simply add a second small cpu to handle critical functions that have realtime needs, and let the Core/Photon firmware deal with the messy network stuff.

I hope that helps! :slight_smile:

Thanks,
David

Is there some sample code such as you describe you can point us at? Thanks.

If I remember correctly, this was being talked about before we had the connection stability patch from TI. Here's one relevant post that I think has code:

Thanks,
David

Hi

I am hitting a similar situation. I need my core code to keep running whatever the state of the WIFI and cloud connection status. All my code needs to run on a regular basis I wrote it on a timer loop system that is non blocking and all parameters are stored in eprom that gets updated by a function. The updates are infrequent and my local software can check status of the connection and report unresponsive. I think I have pretty much all figured except running the core code. I see from this thread that I am probably going to struggle. I saw talk of a watchdog timer but interrupts can only be attached to pins, is there not the possibility of connecting interrupts to internal timers as they are in the arduino?
I am now getting the feeling my development needs to go on hold until March when the Photon comes out :frowning:

Best regards,
Clive

Phew. That’s going to take some time and effort for me to understand. Too much, I’m afraid, as understanding the examples well enough requires me working out quite a lot more of the Spark Core internals then I intend to. Or that I should be expected to. And, bête noire, there’s no docs for things such as IWDG_SYSTEM_RESET.

It seems there is a theme emerging, with quite a lot of people complaining/remarking that they cannot use their Spark as anything more than an Arduino because they cannot get their user code to run uninterrupted/continuously if they are using any WiFi or Cloud (Cloud obviously including WiFi) functionality and the WiFi or Cloud goes away temporarily. My point is that no one is trying to do more than everyone intended to be possible.

I think it would be helpful were Spark.io (the organisation) to bump the issue up the priority list, or say definitively it will not be dealt with and that we should wait for the Photon. I guess(?) the latter is what @Zach wrote above.

1 Like

Hey @psb777 - I totally understand the request for a "final answer" on the topic. Let me do my best to put the issue to rest at least for the time being.

The CC3000 has a blocking host driver. That means that some of the CC3000 calls will block your user code. If that doesn't work with your application, then the Core is NOT the right product for you. The Photon will resolve these issues once and for all, but in the meantime I recommend that you use a second processor or a different product altogether.

The Core has multiple modes of connectivity: MANUAL, SEMI-AUTOMATIC, or AUTOMATIC. If none of these modes works for your application, then you do have the option of digging into the underlying code to make modifications and try and get the behavior you're looking for. The behavior you want may or may not be possible, depending on how much of the particular constraints you're working through have to do with the CC3000 Host Driver code. If you've worked with an Arduino and a CC3000 shield, you'll hit all the same issues.

We are always interested in feedback and suggestions on what we should change. That said, all of our embedded firmware engineers are working on the Photon right now, and will be for the next couple of months, so we're not putting resources to changes on the Core firmware at the moment. As I mentioned earlier, once the Photon has shipped, we will start working on firmware that will affect both the Photon and the Core (because they will both use the same firmware stack, with a Hardware Abstraction Layer to support the differences).

As for a "definitive answer", I can't give one in the long term, because in my opinion that's not how it works. A community member might choose to issue a pull request, which we accept, that changes the behavior of connectivity by changing what some of the connectivity modes do or by providing a fourth option. Or perhaps we will do so ourselves once the Photon is out. But for the next couple of months, we will not be working on the Core firmware stack, which hopefully gives everybody enough clarity to make the necessary architecture decisions for their projects.

5 Likes

Hi All,

I have come up with a temporary solution until my Photon’s arrive. I can run my machines manually with the inbuilt timers and switches. The spark core takes control and allows me to modify the way machines work through the internet.
My solution adds a little more hardware but basically I set up a relay to disable the “in machine” functions this is turned on by the core to allow the core to take control. I control this relay with regular pulses from the core (from my timing loop which is 100ms), if the pulses stop the relay switches back to local machine control until spark core becomes live again. The only time this fails is if it happens to be in the pulse routine when the code stops running, leaving the pin high will continuously enable the relay. I am working on a better solution but hope it can give people ideas on how they may get through this.

Best regards,
Clive

OK, so we’re being forced to wait a Spark generation. Adding an extra CPU to the Spark Core is beyond my skill set. Deciphering the underlying firmware so as to determine undocumented tricks is beyond my patience level.

I think that by the time we get a generation or two further along, when Spark.io announces Photon+=2, then the development/deployment/runtime environment used by that and other IoT devices will be Android/Linux.

I’m redeploying therefore, future proofing my work now for that environment, by using the Raspberry Pi.

I’m nearly finished: Just the central heating relay is still done now by a Spark Core. The thermostats and the controller are Raspberry Pi, the cloud is substituted by exposing the controller Pi on my LAN to the Internet using OpenVPN and a $5/mo virtual server I was already renting for another purpose.

I intended to do all this originally with my three Spark Cores and the Spark Cloud. But when it became apparent that the Spark Core user code would not run should the Cloud or ISP hiccup, then step by step I was forced to abandon the Spark Core. No Cloud required a Pi (or similar) to run the daily timetable. Communicating Pi to Core without the Cloud required UDP, and that was flaky and requires a 15-minute System.reset() reboot.

As an exercise replacing the Spark Core thermostats with Pi’s was trivial, and pleasant. When the relay is similarly Pi connected then I’ll be able to ssh into each device, using cron to generate logs and e-mails etc for each. Oh, and my Pi central heating controller was already the house file server, connected to 2x2TB USB drives.

I think I’ll get the thermostat and relay Pi’s to take part in SETI or protein folding or Mersenne discovery in their spare time. These graphs are Pi produced http://beardsell.com/7Tulk/chgraphs.html and the controller page is virtual server hosted at http://beardsell.com/7Tulk but the work is done on the Pi.

I think Photon+=2 will be able to do all this as well. (And fold proteins too.)

It’s going to be really interesting when you have a power outage or two and suddenly all your pi’s are dead because the SD card got corrupted which is the single most documented fault of the Pis that the Pi foundation has no plans on fixing…

Meanwhile I’ll just be using my Spark powered garage door opener that has been working for months without interruption… through several long power outages and shitty Comcast internet.

Each has their place but Pis are in no way reliable enough to truly depend on.

Adding an extra CPU is as easy hooking up a Teensy or a nano to the Core. If you can string Pi’s throughout your house you should be able to wire up two breadboard compatible microcontrollers

Ah, FUD. When you consider 3million+ Pi’s have been sold that problem is quite infrequently reported, esp nowadays. Go Google looking for number of corruption postings in the last month. https://www.google.co.uk/#q=raspberry+pi+sd+card+corruption&tbs=qdr:m
131 hits and most of those are repeats of old warnings about corruption being possible, not reports of actual corruption.

Paranoid, I do take the precaution of partitioning the SD-card so that there is a /home partition - most writes are not happening to the root partition. The ext4 filesystem is pretty much bullet proof anyway - it’s a journalling filesystem, and half-done writes are rolled back when the power comes back up. Of course, using a good quality SD card is important, there are loads of cheap bad ones for sale. If the SD card stops working …

I have 4 Pi’s and 3 Cores. CFOD? Inexplicably having to use DFU to reflash? Nothing’s perfect.

Oh I know. I just wanted to point out that indeed nothing is perfect. These low cost systems always have some trip ups.

I have personally seen two SD cards (name brands purchased from Amazon) killed in Pis just earlier this year (from power failures) so I know the issue still exists. That being said, the Pi I am using for my whole-house automation system has survived at least 1 unexpected power failure so it obviously doesn't happen every time.

1 Like