Continuous HTTP requests

Hello guys @Spark,

We are sending HTTP requests continuously from the Spark to a web server (Ubidots), but sometimes the Core just stops sending requests, then flashes red rapidly and then restarts. This happens after ~40-50 loops.

I thought it might be Internet issues, but another user reported the same problem on different networks.

I wrote this code to see after how many requests it fails:

#include "dnsclient/dnsclient.h"
#include "HttpClient/HttpClient.h"

HttpClient http;
char resultstr[64];

#define VARIABLE_ID "5499ba207625421efc176fdc"
#define TOKEN "j2aawKrPRmPRszEbwauSlnJEA2xxxx"
int lightLevel = 0;

http_header_t headers[] = {
    { "Host", "things.ubidots.com" },           
    { "Content-Type", "application/json" },
    { "X-Auth-Token" , TOKEN },
    { NULL, NULL }                              
};

http_request_t request;
http_response_t response;

IPAddress dnsServerIP(8,8,8,8);
IPAddress remote_addr;
DNSClient dns;
char serverName[] = "things.ubidots.com";

void setup() {     
    pinMode(A0, INPUT);
    request.port = 80;
    dns.begin(dnsServerIP);
    dns.getHostByName(serverName, remote_addr);
    request.ip = remote_addr;
    request.path = "/api/v1.6/variables/"VARIABLE_ID"/values";
    Serial.begin(9600);
}

void loop() {     
    lightLevel++;
    request.body = "{\"value\":" + String(lightLevel) + "}";
    http.post(request, response, headers);
    Serial.println(response.status); 
    Serial.println(response.body);
}

Is there a known issue when sending too many requests? or is there a need for a “flush” type of command?

Thanks in advance!

1 Like

Same thing happen to mine, I actually did 5 second delay in between sending, but still have the same issue.The core keeps droping out. Any idea would be appreciated.
Thanks

The red flashes you see are an error code and knowing the code might help. They are documented here:

http://docs.spark.io/troubleshooting/#troubleshoot-by-color-flashing-red

The user-contributed HTTP library you are using does use Arduino String objects which do use heap memory and don’t always free it. Garbage collection on the core is not perfect. In your loop() you make a String of the lightLevel which consumes some bytes every time you go around the loop. You might get it to work for longer by using sprintf() into char array instead.

1 Like

I didn't know this - do you have any examples? Is it because of a fragmented heap?

There have been a bunch of folks here in the forum that have gotten 8 flashes for out of heap using Strings, particularly when concatenating together substrings, since the substring operator returns a new String object every time. Getting those folks to move to a more static allocation scheme either with Strings or char arrays has fixed the problem.

There have been lots of problems reported with String in Arduino land as well.

I think they are still OK to use, but if you are having memory problems, it is a good thing to look at.

3 Likes

Thanks @bko, I didn’t know this either.

@ruipengt let’s try this then:

When declaring the variable:

float lightLevel = 0.0;
char resultstr[64];

When building the request body:

sprintf(resultstr, "{\"value\":%.4f}",lightLevel);
request.body = resultstr;

This part does not do what you want:

request.body = resultstr;

This copies the address of the char array resultstr in to the request.body.

Because the HTTP library expects Strings, you will need one, but the trick is you can copy into the already allocated internal buffer

float lightLevel = 0.0;
char resultstr[65]; //allow for trailing zero terminator
// construct a String with 64 chars to reserve space.
String bodyString("1234567890123456789012345678901234567890123456789012345678901234");

sprintf(resultstr, "{\"value\":%.4f}",lightLevel);
for(int i=0;i<64;i++) {
  bodyString.setCharAt(i,resultstr[i]);
}

You need to be a little careful if you change the sizes not to run off the end of things. There is actually a way to do this using the String c_str that avoids having to have a separate char array, but it is a little more dangerous.

Thanks very much for the response, I actually tried the below code and it worked. looks like passing a string pointer to request.body is ok. I think the memory allocation problem indeed caused the core to crash. So I’ll keep using static array for the storage. Thanks again. I’ll keep an eye on the core to see how long it last.

sprintf(resultstr, "{\"value\":%.4f}",lightLevel);
request.body = resultstr;
1 Like

@mdma Does the firmware use any dynamic memory?

At present, not much, but that will change as the firmware team move much of the internals over from static to dynamic allocation, so you only pay for what you use, and will remove some hard-coded limits.

String uses dynamic memory - I would be surprised if there is memory leak as such (memory being allocated and not freed) since the code has been extensively peer-reviewed, but I can imagine that

  1. more memory is allocated than might be expected, with intermediate strings each being allocated memory
  2. the high number of allocations and frees, coupled with the relatively limited heap size on Arduino/Spark Core will mean heap fragmentation becomes an issue relatively quickly.

Heap fragmentation is when there is free space available on the heap, but none of it is large enough to satisfy the current request. This can happen after a number of allocations and frees. For example, after a number of allocation/free cycles, the heap might look like this:

[ 32-bytes free ][ 8-bytes used ]....

…all the way through. So, although the heap is 80% free, attempting to allocate a String larger than 32 bytes will fail.

In general, there’s not much that can be done with heap fragmentation. But in specific cases we can make the situation better. For example, if the 8-bytes above were Strings, the string objects could work together to reallocate their space, starting with the ones lowest in memory - this would cause the first 8-byte allocated block to be released, reallocating at the start of the free 32-byte block, making a region of 64-bytes. Continuing this through all strings would result with all 8-byte regions being allocated at the start of memory, and all free space at the end.

One way to avoid heap fragmentation is to avoid many allocations of which all but a few are freed. As @bko mentions, this is precisely what happens when using String to do concatenation - many intermediate strings are constructed, followed by the final string, and then the intermediate strings are freed, while the final string is retained - classic recipe for heap fragmentation. The solution here is to use classic C char buffers and sprintf/strcat to build the string

Btw, on the photon, heap fragmentation will be less of an issue for the same applications since the heap is much larger. \o/

3 Likes

At present, where is this memory used? Could it be a source of problems with the TCP/IP stack? -- Well I'd guess not, the TCP/IP stack problems are more likely how the Spark works with the CC3000 TI Driver.

I'm starting to convenience myself that there is NOT any Stable Spark configuration. Cloud / No cloud. all my tests have indicated problems with the networking, the Spark sometimes recover using a watchdog timer, but that is not a reliable solution as there are long delays between reentry of the user loop.

I have noticed that the RSSI signal strength goes to 2dB when the CC3000 starts to exhibit problems. After 10's of minutes something will cause it recover and it will go back to -70dB. It does not matter what access point I use (I have several) so I am suspecting its with the CC3000.

It sure would be nice to get a stable CC3000 configuration such that we can leverage the networking of the Spark.

Maybe can get a sticky thread going where we can make some progress towards this problem? There are several threads on this topic over the past year.

Just as a counterpoint to this, I have a weather reading app that scrapes a web page via TCPClient for local weather forecasts and displays them on a 2 line LCD. This sits on my desk and just runs and runs. I would say it crashes around once every week or two with flashing cyan, but generally recovers without intervention.

I have another core that runs my NTP library with an LED clock and temperature display using UDP. It also runs and runs, seemingly never crashing.

Sometimes, probably once a month or so, the cloud or my internet goes down and both cores start to flash to cyan and need to be reset if they don’t recover. I have a very typical home broadband internet connection with great reliability.

The TI CC3000 8-bit register with the RSSI has a 7-bit RSSI and another one bit field, so a +dB value indicates a failure of some kind in the TI part.

2 Likes

Thanks @bko, it’s working now for me as well. I will test your code too and record the results. I still get some packets lost but I want to measure exactly how much percentage that is.

@bko,
I agree that the core does not crash very often (unless you are trying to use the TCPServer), it does however get into a state where the networking no longer works and eventually recovers. I would guess your NTP and Web Page scraping apps do fail. Is it possible you just are not aware of it unless you are logging the failures, or the failures are occuring outside of your intervals for scraping or NTP syncing?

They do fail at the periods I mentioned: a week to a month or so, unless the cloud or my whole internet goes goes down, which I would say is roughly monthly. I am not currently logging but when I wrote my Spark.publish tutorial with “uptime” I did gather days and days of data with no failures publishing every 15 seconds. I had a lot of data in a spreadsheet where I was analyzing timestamps versus millis() time to try to learn more about cloud latency at the time (which for the record seems pretty constant at around 138ms from my house).

I do think that local network conditions have an effect on the stability of your connections. I have my Sparks on a separate router that does not get a lot of other traffic. Because of this I never saw the ARP problems that the Deep Update TI patch tries to fix, for instance. People here in the forum with complicated or slow networks just seem to have more trouble.

The TI part seems to have some timeouts built-in around 6 seconds long, so if you try to connect to a nonexistent host, it will ARP for that host for about 6 seconds before giving up, for instance. Some people have thought that was broken behavior, but I really can’t fault the core on that one. If you are checking every return value from every TCP and UDP call, you can recover, but not any quicker than about 6 seconds. You asked it to do something and it tried hard for 6 seconds before giving up, is how I read that.

Finally, I certainly agree that TCPServer does not work with high-speed data in volume! Which is strange because it is a pretty thin veneer over the client code. I don’t have an explanation for that.

As I have said before, I advocated with the Spark guys for the next generation Spark (now Photon) to have a WiFi chip that was more, shall we say, professional. I have great hope for the new Photon and I think it will be much more stable just because lots of other devices have used the WiFi chips in Photon successfully.

If it was just about "high-speed data in volume" I would be thrilled, but that is not the case. Just sending 200 bytes of data every 2 seconds will cause the CC3000 to go away after about 10-20 minutes and it will never recover. If there was a way to detect it's failure maybe I could manually reset it by turn off CC3000 and restarting it.

I agree that we are all hoping the Photon will be a production quality product. I have a few products waiting for the Spark to become production ready (not just for hobbyist) and am hopeful the Photon will give the products a chance in the market.

WRT to your NTP and Web Scraping apps, how do you know if they miss a window? For example I have an app that reports data every 10 minutes, however it will often fail within the 10 minute intervals and recover before the next data reporting time. Unfortunately I need the network to work properly inside of the 10 minute intervals. The only way I can tell it's in a funky state is by my loop LED pulse not occuring every 1/2 second.

With the weather app, it clears the display before it goes to get a new update, so you know it failed when the display is blank. With the clock, it stops updating the flashing colon between hours and minutes.

@bko, thanks for the response, Can you tell me how often does each run and for the web app how much data does it process from the network?

Trying to get my head around what’s going on w/ my app.

Hi @mtnscott

The web scrape app reads around 2700 bytes every 5 minutes. It reads and parses the data one byte at a time without ever storing much but the displayed data.

The NTP library updates every hour which is just a pair of really small UDP 48-bytes packets. The source for this one is available here:

1 Like